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ABSTRACT 


This  thesis  presents  an  analysis  of  the  application  of  the  Modular  Command  and 
Control  Evaluation  Structure  (MCES)  to  the  Identification  Friend,  Foe.  or  Neutral 
(IFFN)  Joint  Testbed.  The  MCES  and  IFFN  Testbed  evaluation  approaches  are  also 
compared.  MCES  is  a  structured  approach  to  evaluate  Command  and  Control  (C2) 
systems  which  uses  standard  and  evolving  operational  research  tools.  The  MCES 
approach  provided  the  IFFN  Joint  Testbed  with  an  air  defense  C2  system  architecture 
which  became  a  descriptive  tool  for  C2  analysts  to  define  and  evaluate  measures  to 
determine  the  effectiveness  of  competing  air  defense  C2  systems.  This  IFFN 
application  served  as  an  evaluation  and  refinement  of  MCES  as  well  as  a  tool  for 
assisting  the  IFFN  Joint  Test  Force  in  evaluating  U.S.  air  defense  C2  systems  in  the 
NATO  area. 
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I.  INTRODUCTION 


A.  NATURE  OF  THE  PROBLEM 

How  much  of  a  force  multiplier  can  be  attributed  to  a  particular  command  and 

control  (C*')  systems?  Given  several  alternative  systems,  which  is  best?  What  has 

to  be  measured  to  determine  this  difference?  Are  all  the  relevant  factors  taken  into 

account?  These  are  complex  C“  questions  being  asked  by  the  Joint  Chiefs  of  Staff  as 

well  as  other  senior  military  commanders  as  they  are  faced  with  acquiring,  testing,  and 
_■> 

operating  C“  systems. 

A  methodology  is  needed  to  describe  the  C  systems  architecture  which  will  allow 
analysts  to  measure  systems  response  and  attribute  the  effectiveness  of  that 
response  to  the  elements  or  structural  relationships  which  form  that  system.  There 
is  a  definite  need  for  generic  tools  to  evaluate  systems  and  architectures.  What  was 
lacking  in  current  evaluation  methodologies  was  a  method  to  relate  systems  to 
measures  of  its  contribution  to  force  effectiveness  and  mission  accomplishment.  In  the 
past,  evaluation  has  been  conducted  in  a  piecemeal  fashion  with  assorted  evaluation 
tools  only  for  specific  parts  of  the  problem.  The  Joint  Chiefs  of  Staff  Command. 
Control  and  Communications  Systems  Directorate's  (JCS  C3S)  recognized  these  needs 
and  required  development  of  a  paradigm  to  evaluate  competing  C~  architectures 
[Ref.  1:  p.  8).  The  Modular  Command  and  Control  Evaluation  Structure  (MCES) 
attempts  to  address  this  need. 

B.  MCES  METHODOLOGY 

MCES  was  developed  as  a  structured  approach  to  evaluate  C^  systems  and  uses 
standard  and  evolving  operations  research  tools.  MCES  attempts  to  integrate  the 
previous  efforts  of  C^  users  and  analytic  organizations  to  form  a  single  C"  evaluation 
package. 

The  MCES  is  composed  of  seven  separate  modules  which  guide  analysts  through 
the  command  and  control  evaluation.  Figure  1.1  represents  the  seven  modules  of  the 
MCES  methodology  and  the  output  from  each  module.  The  first  module  is  used  by 
the  analyst  and  operational  user  to  define  the  particular  C^  problem.  The  next  three 
modules  set  the  terminology  and  theory  to  describe  the  C^  system  architecture  which 
permits  analysts  to  modei  the  C*  system  and  its  operation.  Inherent  in  the 


is 


elements  and  process  functions  into  a  structural  framework.  .VICES  shares  common 

I  ^  \  .  1.  . — 1  . .  .  •  n  f  ^  pt-pfor*.  r  AlP^ln4llP\n  PiMaf  Urt/1  nlrtrt.ap  I  Ua  \  !  1  I '  C  .  Id  I  ipa^p 


terminology  of  current  C  systems  evaluation  methodologies.  The  MCES  defines 
"architecture"  as  a  description  of  an  integrated  set  of  systems  whose  physical  entities, 
structure  and  functions  are  coherently  related.  The  architecture  provides  a 
representation  which  will  eventually  lead  to  the  ability  to  measure  the  C“  system 
response  and  the  effectiveness  of  directing  forces  to  accomplish  the  mission.  The  C" 
theory  behind  these  modules  is  robust  enough  to  allow  analysts  to  reconfigure  the 


particular  O  system  physically  or  structurally  within  the  architecture  during  the 


evaluation.  In  module  5,  measures  are  developed  which  will  be  used  to  discriminate 


between  alternative  configurations  of  the  architecture.  When  the  measures  for  the  C“ 


system  have  been  identified,  the  sixth  module  requires  that  a  suitable  data  generator  be 
selected  or  developed  to  derive  the  values  for  the  measures  selected.  The  final  MCES 
module  is  used  to  aggregate  and  evaluate  the  measure  results  in  order  to  determine 
the  optimal  system  for  a  particular  mission.  [Ref.  1:  pp.  10-23] 

C.  MCES  EVOLUTION 

The  Modular  Command  and  Control  Evaluation  Structure  (MCES)  was 
developed  at  the  Naval  Postgraduate  School  with  research  support  from  the  Military 
Operations  Research  Society  (MORS)  and  monetary  support  from  different  military 
agencies  [Ref.  2:  p.  13].  The  MCES  development  started  with  military  community 
research  and  discussion  concerning  the  need  to  develop  and  quantify  measures  of 
effectiveness  appropriate  to  C^  systems.  According  to  the  MCES  principal 
investigator,  Dr.  Ricki  Sweet,  MCES  is  evolving  as  any  scientific  development  in  the 
following  steps  [Ref.  1:  p.  31]: 

1.  Public  discussion  and  mandate  for  clarification; 

2.  Setting  up  the  nature  of  the  problem,  the  tools,  definitions,  and  potential 
directions; 

3.  First  order  development  of  the  identified  components; 

4.  Specification  of  the  interrelationships  of  the  components; 

5.  Testing  of  the  theory  with  real  problems,  i.e.,  extra-laboratory  experiments;  and 

6.  Refining  the  structure  in  accordance  with  the  test  results. 

The  MCES  methodology  is  evolving  in  order  to  resolve  key  C~  issues. 
Throughout  this  evolution,  C^  tools  and  models  have  been  identified,  developed,  and 
integrated  into  the  MCES.  Having  completed  the  first  cycle  of  the  referenced  six  steps 
of  scientific  development,  MCES  is  now  in  the  process  of  a  continuing  iteration  of  the 
last  two  steps  of  test  and  refinement.  The  Identification  Friend,  Foe  or  Neutral 
(IFFN)  Joint  Testbed  application  is  an  example  of  such  a  MCES  test  and  refinement 
iteration.  This  scientific  development  will  lead  to  a  further  refined,  bounded  and 
generic  methodology  that  may  fulfill  many  C^  architecture  evaluation  requirements. 

D.  IFFN  BACKGROUND 

The  IFFN  testbed  is  currently  addressing  the  air  identification  problem,  which  is 
a  subset  of  the  overall  air  defense  C^  problem.  The  testbed  is  representative  of  the 
NATO  air  defense  C^  system  which  must  operate  in  an  environment  of  friendly,  enemy 
and  neutral  aircraft  to  perform  its  air  defense  mission.  Within  the  air  defense  Cu 
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architecture,  geographically  separated  radars  and  special  intelligence  sources  develop 
detection,  track,  and  identification  information  on  air  objects.  Computers  are  then 
used  to  store  and  correlate  this  information.  Digital  communications  are  then  used  to 
share  the  track  information  between  command  facilities.  The  command  organizations 
develop  perceptions  of  the  battle  situation  and  make  decisions  to  achieve  mission  goals 
based  on  these  perceptions.  The  command  organization  then  implements  the  decisions 
by  directing  and  controlling  air  defense  forces  to  take  some  action  against  the  enemy 
forces.  The  test  concept  uses  a  computer  simulation  of  manned  and  simulated 
command  centers  and  weapon  systems  employing  real  world  operating  procedures 
against  varied  threat  scenarios.  [Ref.  3:  pp.  3-5] 

The  IFFN  system  bounds  were  defined  by  geographic  areas  of  responsibility 
within  the  NATO  Fourth  Allied  Tactical  Air  Force  (4ATAF)  sector.  The  specific 
command  centers  that  perform  the  functions  are:  Sector  Operations  Centers 
(SOC),  Control  and  Reporting  Centers  (CRC),  Brigade  Fire  Direction  Centers  (Bde 
FDC),  and  Battalion  Fire  Direction  Centers  (Bn  FDC).  Information  sources 
considered  to  be  within  the  C^  system  are:  NATO  Airborne  Early  Warning  systems 
(NAEW),  Special  Information  (intelligence)  Systems  (SIS),  and  other  information 
sources  (i.e.,  (light  plans).  The  air  defense  C^  system  architecture  included  the  weapon 
systems  when  they  performed  C^  functions.  The  air  defense  weapon  systems 
considered  by  the  IFFN  testbed  are  the  F-15,  all  weather  fighter,  and  the  HAWK  and 
PATRIOT  surface-to-air  missiles  (SAMS). 

E.  MCES  APPLICATION  TO  THE  IFFN  TESTBED 

A  MORS  workshop  team  specifically  researched  the  IFFN  problem  during  a 
conference  to  assist  the  IFFN  Testbed  in  finding  a  solution  to  the  IFFN  problem.  The 
initial  C“  problem  statement  formulated  by  the  January  1985  MORS  Workshop  from 
the  original  IFFN  Testbed  issues  was: 

How  effective  is  the  Central  Europe  air  defense  C2  svstem  in  providing  decision 
makers  the  means  to  assess  the  situation  and  emplov  air  defense  assets  in  order 
to  meet  overall  mission  objectives?  [Ref.  4:  p.  1] 

During  the  1985  MORS  C^  Evaluation  Workshop,  the  IFFN  Test  Director,  Colonel 
Dave  Archino  issued  the  following  challenge  to  the  working  group: 


Develop  a  tool  .  .  .  specific  to  air  defense  that  allows  IFFN  to  evaluate  the  flow 
of  C2  information  throuehout  the  C2  structure  and  determine  if  it  is  useful  or  not 
in  winning  the  war  .  .  .  meeting  the  mission  objectives  .  .  .  and  operational  issues 
IFFN  plans  to  address.  [Ref.  4:  p.  1J 

It  was  determined  that  MCES  could  be  tailored  to  help  solve  the  IFFN  testbed 
requirements.  Major  Patrick  Gandee,  while  a  Naval  Postgraduate  School  student,  was 
the  principal  investigator  of  the  MCES  application  to  the  IFFN  Testbed. 

F.  THESIS  ORGANIZATION 

This  thesis  will  summarize  how  the  MCES  was  specifically  applied  to  the  air 
defense  problem  and  how  that  application  has  been  used  by  the  IFFN  testbed  to 
address  their  operational  issues.  A  comparison  of  the  IFFN  Testbed  and  MCES 
approaches  will  be  presented.  Since  MCES  was  concurrently  evaluated  and  refined  in 
the  IFFN  application  process,  this  thesis  will  continue  the  evolution  process  by 
formulating  recommendations  for  further  MCES  refinement. 

The  MCES  methodology  will  be  outlined  in  Chapter  II.  The  IFFN  Testbed  and 
its  evaluation  approach  will  be  described  in  Chapter  III.  Chapter  IV  will  describe  the 
application  of  MCES  to  the  IFFN  Testbed.  A  discussion  of  the  differences  between 
the  MCES  and  IFFN  Testbed  approach  is  presented  in  Chapter  V.  Conclusions  and 
recommendations  concerning  both  the  IFFN  Testbed  and  the  MCES  methodology  are 
presented  in  Chapter  VI. 


II.  MCES  METHODOLOGY  DESCRIPTION 


A.  INTRODUCTION 

The  following  description  of  MCES  is  taken  in  part  from  Dr.  Sweet's  report  on 
MCES  [Ref.  1:  pp.  10-23]  and  from  her  notes  and  briefings.  The  figures  presented  in 
this  chapter  are  revised  and  updated  versions  of  the  ones  she  used  in  her  publications 
and  briefings.  A  more  detailed  description  and  analysis  of  MCES  will  be  presented  in 
Chapter  IV  when  the  MCES  application  to  the  IFFN  testbed  is  used  as  an  example. 

B.  MCES  MODULES 

MCES  is  divided  into  seven  modules  which  are  detailed  below  by  module. 

1.  Module  1:  Problem  Formulation 

In  module  1,  the  decision-maker's  analysis  objectives  and  needs  are  described 
for  a  specific  C^  problem.  First,  the  decisionmaker's  needs  are  characterized.  The 
analysts  consider  the  decisions  being  formulated,  assumptions  about  the  problem,  the 
level  of  analysis  required,  and  the  mission  supported.  Both  the  appropriate  scenarios 
and  assumptions  underlying  the  evaluation  are  made  explicit  and  the  required  level  of 
analysis  is  determined.  This  problem  statement  is  then  used  in  the  second  module  to 
bound  the  C2  system  of  interest.  The  implementation  of  this  module  results  in  a  more 
precise  statement  of  the  problem  and  analysis  objectives.  Figure  2.1  lists  the  major 
actions  required  in  module  1.  [Ref.  1:  pp.  10-11] 

2.  Module  2:  C^  System  Bounding 

Module  2  identifies  the  relevant  system  elements  that  will  bound  the  system. 
When  bounding  the  C^  system,  a  three  component  definition  of  a  C2  system  is  used 
based  on  JCS  Publication  I  [Ref.  5:  p.  77],  These  definitions  state  that  a  C^  system 
consists  of  physical  entities,  structures,  and  C^  processes.  Physical  entities  are 
equipment,  software,  people  and  their  associated  facilities.  Structure  includes 
organization,  procedures,  protocols,  concepts  of  operation  and  information  flow 
patterns.  The  term  "C^  process''  refers  to  what  the  system  is  doing  or  the  functions 
that  the  process  performs.  Bounding  the  system  requires  bounding  of  the  physical 
entities  and  structure.  The  processes  are  developed  in  Module  3. 

There  are  two  issues  that  are  raised  in  this  module.  The  first  issue  is  the 
mapping  of  system  physical  components  and  personnel  to  the  systems  boundaries. 
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Figure  2.1  Module  1,  Problem  Formulation. 


The  second  issue  concerns  determining  the  required  levels  of  analysis.  One  method  of 
graphically  illustrating  this  process  of  bounding  the  environment  is  through  the  use  of 
Dr.  Sweet's  "onion  "  as  a  representative  of  the  environment.  The  insert  of  Figure  2.2 
displays  this  representation.  Starting  in  the  middle  of  the  onion  are  the  subsystems  of 
the  C*  system.  Going  out  from  the  middle,  the  Ct  subsystems  constitute  the  C2 
system.  The  system  is  itself  a  component  of  the  overall  force  which  in  turn  is  a 

part  of  the  environment.  The  area  outside  of  the  "Onion  Skin"  is  the  rest  of  the  world. 
Successive  "peeling"  of  the  "onion  skin"  will  revel  the  subsystems  to  be  evaluated. 
Module  2  results  in  the  bounding  of  the  problem  and  ihe  identification  and 
categorization  of  the  system  elements  of  physical  entities  and  organizational  structure. 
Figure  2.2  depicts  the  activities  for  Module  2.  [Ref.  1:  pp.  12-13] 

3.  Module  3:  Process  Definition 

Module  3  defines  the  processes  needed  to  fulfill  the  mission.  The 
particular  command  and  control  process  is  described  by  analyzing  the  generic  C* 
processes  of  the  system.  The  proposed  MCES  solution  to  understanding  the  C‘ 
processes  of  a  particular  system  is  to  use  an  information  based  paradigm  similar  to 
the  J.  Lawson  C“  Process  Model  (Ref.  6:  pp.  93-99]  in  the  broader  framework  of 
MCES.  The  insert  of  Figure  2.3  displays  a  modified  Lawson  s  generic  C*  process  loop 


Figure  2.2  Module  2,  System  Bounding. 


model.  One  example  of  Lawson's  generic  C‘  loop  model  components  is  the  ASSESS 
block.  The  assessments  for  human  decisions  are  made  by  battle  commanders 
performing  the  ASSESS  function.  The  commanders'  assessments  are  made  by 
perceiving  and  assigning  meaning  to  the  overall  situation.  Commanders  use 
information  from  their  own  sensors,  feedback,  from  their  forces,  and  an  interface  with  a 
separate  intelligence  process  to  develop  these  perceptions  about  the  enemy  and  friendly 
capabilities.  The  other  functions  of  Lawson's  generic  process  model  are  performed 
in  most  processes.  [Ref.  1:  p.  14) 

It  is  necessary  to  provide  a  translation  of  the  vocabulary  of  the  problem 
into  the  terminology  of  the  generic  process  model  to  effectively  use  MCES.  This 
translation  of  terms  helps  the  analyst  keep  from  overlooking  critical  processes  and 
provides  standard  vocabulary  and  definitions.  In  the  past,  MCES  has  been  able  to 
apply  the  generic  C^  process  model  successfully  with  only  minor  modifications.  There 
are  different  C^  process  levels  and  interactions  among  the  C^  process  function 
components  and  these  process  relationships  can  become  very  complex.  To  illustrate 
different  levels  of  processes,  an  example  of  a  commander  performing  a  decision 
function  can  be  used.  The  commander  passes  his  decision  to  several  subordinates  who 
in  turn  work  out  detailed  instructions  to  implement  that  decision.  These  subordinates 
communicate  the  instructions  to  their  forces  which  then  act  in  the  environment.  In 
command  and  control  terms,  the  commander  and  the  subordinates  were  performing 
separate  decision  functions  within  a  process.  The  subordinates'  decision  function  is 
related  to  the  commander's  decision  function  by  the  commander's  decision  (output) 
and  the  subordinates'  receipt  (input).  In  turn,  the  detailed  instructions  from  the 
subordinates  to  the  force  couple  the  subordinate's  function  to  the  force  function.  This 
functional  input;  output  relationship  forms  a  "structure"  between  separate  process 
functions  which  are  required  to  perform  the  mission.  The  structure  determines  the 
information  flow.  [Ref.  1:  p.  15] 

In  a  distributed  system,  processes  may  be  related  to  other  processes.  The 
processes  that  have  been  valuable  to  MCES  evaluations  for  describing  distributed 
information  flow  are  [Ref.  1:  pp.  70-73): 

1.  Intelligence  (INTELL)  Process.  Assigns  meaning  to  observed  activities  and 
situations  and  forecasts  changes  in  the  current  situation. 

2.  Crosstell  (XTELL)  Process.  A  subset  of  the  communications  process,  which 
provides  for  sharing  of  information  throughout  the  C2  svstem  to  support 
decisions  and  their  implementation. 

3.  Execution  Level  C2  Process.  Directly  controls  weapon  systems. 
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Figure  2.3  Module  3,  C"  Process  Definition. 


A  complete  picture  of  the  C~  systems  architecture  includes  the  intelligence  process  and 
how  it  interfaces  with  the  C“  process.  XTELL  is  the  sharing  of  information  and  is 
needed  to  describe  the  coordination  between  distributed  systems.  At  a  single 
command  node,  XTELL  is  simply  a  process  function  but  on  a  systems  level  it  is  a 
networking  process.  At  a  single  node,  the  process  directly  controls  weapon 
systems.  At  a  higher  level,  the  C“  process  coordinates  the  directing  of  the  weapon 
systems.  These  processes  are  dynamic  descriptions  of  what  the  C‘  system  is  doing. 
XTELL  and  IXTELL  processes  interfaced  with  the  process  in  a  distributed  system 
are  ultimately  linked  to  the  weapon  systems  which  perform  the  mission. 

Dr.  Sweet  emphasizes  that  this  module  forces  the  analysts  attention  on 
[Ref.  1:  p.  14J: 

1.  the  environmental  cause  or  initiator  (i.e.,  the  enemy  force)  of  the  C2  process 
that  results  from  a  change  in  the  desired  state; 

2.  the  internal  C2  process  functions  that  characterize  what  the  svstem  is  doing 
such  as  sense,  assess,  generate,  select,  plan,  and  direct;  and 

3.  the  input  and  output  from  the  internal  C2  process  that  couples  with  the  force 
process 

Figure  2.3  represents  the  major  actions  required  in  module  3.  As  a  result  of  the 
implementation  of  this  module,  the  functions  of  the  process  for  a  given  problem  are 
identified  and  mapped  to  the  generic  process  loop. 

4.  Module  4:  Integration  of  System  Elements  and  Functions 

Module  4  relates  the  information  flow  to  a  system  by  means  of  its  C“ 
process  functions.  Functions  are  subsets  of  the  process  and  represent  what  the  C2 
system  actually  does  or  accomplishes.  The  relationship  of  the  physical  entities  to 
the  process  functions  and  organizational  structure  is  also  formulated  in  this  module. 
This  integration  is  accomplished  by  making  explicit  the  relationships  between  these 
components.  Figure  2.4  outlines  the  actions  required  in  Module  4. 

The  first  step  is  to  map  the  physical  entities  of  man  and  machine  which 
perform  the  functions  and  produce  output  to  an  organizational  structure.  All  C“ 
functions  can  be  potentially  performed  in  a  single  node  or  be  distributed  between 
different  nodes  so  this  mapping  results  in  an  organizational  structure  which  graphically 
depicts  a  single  node  or  a  distribution  of  command  and  weapon  nodes  depending  on 
the  system's  unique  configuration. 

Next,  the  flow  of  information  is  charted  by  techniques  such  as  Data  Flow- 
diagrams  (DFDs)  [Ref.  7:  pp.99-1 15}  or  Petri  Nets  [Ref.  8]  which  may  be  used  to 
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Figure  2.4  Module  4,  Integration  of  System  Elements  and  Functions. 

describe  information  flow  through  the  C~  process  model.  DFDs  and  Petri  Nets  will  be 
discussed  in  greater  detail  in  Chapters  V  and  VI.  DFDs  are  an  example  of  a  technique 
that  has  been  productive  in  this  module  in  determining  relationships  between  processes 
and  information  How.  Data  Flow  Diagrams  (DFDs)  are  first  constructed  to  show 
information  flow  through  the  process  model.  The  inputs  and  outputs  of  each 
function  are  determined  and  related  to  the  other  functions  in  the  process.  In  the  next 
step,  a  transform  analysis  is  performed  to  uncover  the  transform  center  (process  center) 
and  to  determine  the  subordinate  and  superordinate  relationships  between  the 
transform  center  and  the  individual  functions.  Information  flows  into  the 
transform  center  and  control  information  flows  out  of  the  transform  center.  These 
input  output  relationships  describe  the  internal  information  How  between  separate 
process  functions.  The  data  flowing  into  the  transform  center  or  process  center  is 
information  flow  while  the  data  flow  out  of  the  transform  center  is  control  information 
flow  in  the  form  of  action  requests  or  commands.  Thus  a  hierarchical  "structure  has 
been  defined  in  terms  of  the  mission  essential  information  flow  between  functions 
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within  the  C“  process.  From  this  information,  a  C"  system  architecture  will  eventally 
be  formulated.  These  procedures  will  result  in  the  description  of  how  the  element*;  and 
players  function  together,  which  is  basically  relating  information  How  to  organizational 
structure.  [Ref.  1:  pp.  16-17] 

5.  Module  5:  Specification  of  Measures 

Module  5  specifies  the  measures  necessary  to  evaluate  the  C“  problem.  These 
measures  are  then  classified  as  to  their  level  of  measurement,  i.e.,  dimensional 
parameters,  measures  of  performance  (MOPs),  measures  of  effectiveness  (MOEs).  or 
measures  of  force  effectiveness  (MOFEs).  MOFEs  are  used  to  describe  the  actions 
between  the  force  and  the  environment.  When  the  system  is  combined  with  the 
force,  the  environment  will  be  effected  and  MOFEs  measure  this  force  effect.  Within 
the  force  boundary,  MOEs  are  used  to  measure  how’  the  combat  force  is  effected  by  the 
CL  operation.  MOPs  are  applied  at  the  C  system  boundary  and  measure  how  well  the 
system  performs  its  functions.  For  the  subsystem  within  the  boundary  of  the 
system,  dimensional  parameters  are  used  to  measure  the  limits  of  the  subsystems.  The 
resulting  measures  may  be  used  to  determine  differences  in  a  system  when  utilizing 
alternative  configurations  of  the  physical  entities,  structure,  or  processes.  Figure  2.5 
graphically  depicts  the  "onion"  with  its  corresponding  measures  and  the  actions 
required  in  Module  5. 

This  MCES  implementation  results  in  the  specification  of  a  set  of  measures 
focused  primarily  on  the  process  functions.  The  process  functions  identified  may  be 
used  to  derive  a  complete  set  of  relevant  measures  which  can  then  be  subjected  to 
further  scrutiny.  A  set  of  measures  can  be  compared  to  a  set  of  desired  measures 
characteristics  as  shown  in  Table  1  [Ref.  1:  p.  20]  to  insure  that  the  measures  are 
useable. 

6.  Module  6:  Data  Generation 

In  Module  5,  one  of  several  types  of  data  generators  such  as  exercises, 
experiments,  simulations,  subjective  judgements,  or  relevant  experiences  is  selected  to 
generate  the  necessary  values  for  the  measures  formulated.  These  values  may  be  either 
measured  directly  or  indirectly.  The  analysts  consider  the  reproduciblity  of  results, 
precision  and  accuracy,  timing  of  collection,  environmental  controls,  and  experimental 
design  during  this  module.  A  timeline  is  formulated  to  set  the  completion  dates  for  the 
data  generation  phases.  Using  the  designated  data  generator,  the  resulting  values  for 
these  measures  constitute  the  output  of  this  module.  Figure  2.6  outlines  the  data 
generation  module.  [Ref.  1:  p.  21] 
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Figure  2.5  Module  5,  Specification  of  Measures. 


TABLE  1 

CRITERIA  FOR  EVALUATION  MEASURES 


CHARACTERISTICS 

DEFINITION 

Mission  oriented 

Relates  to  force/system  mission. 

Discriminatory 

Identifies  real  difference  between 
alternatives. 

Measurable 

Can  be  computed  or  estimated. 

Quantitative 

Can  be  assigned  numbers  or  ranked. 

Realistic 

Relates  realisticly  to  the  C2 
system  and  associated  uncertainties. 

Objective 

Can  be  defined  or  derived, 
independent  of  subjective  opinion. 

Appropriate 

Relates  to  acceptable  standards  and 
analysis  objectives. 

Sensitive 

Reflects  changes  in  system  variables. 

Inclusive 

Reflects  those  standards  required 
by  the  analysis  objectives. 

Independent 

Is  mutually  exclusive  with  respect 
to  other  measures. 

Simple 

Is  easily  understood  by  the  user. 

7.  Module  7:  Aggregation  of  Measures 

Module  7  is  the  final  module  and  addresses  the  issue  of  the  aggregation  and 
interpretation  of  the  observed  values  of  the  measures.  Figure  2.7  depicts  the 
aggregation  and  interpretation  process.  From  data  generation,  values  for  the  identified 
measures  will  be  obtained  and  analyzed.  One  of  the  analysts'  concerns  will  be  to  relate 
command  and  control  systems  to  some  measure  of  force  effectiveness  which  is 
sometimes  termed  the  force  multiplier  effect.  For  MOFEs,  the  intent  of  aggregation  is 
to  relate  the  system  with  combat  systems  to  indicate  combat  outcomes.  After 
aggregation,  the  issues  of  measure  causality,  sufficiency,  and  independence  are  to  be 
considered.  Scenario  dependence  must  also  be  addressed.  Because  combat  is  very- 
complex,  many  measures  will  not  show  significant  differences.  Analysis  must  be 
conducted  to  determine  the  important  factors  in  the  particular  scenarios.  At  this 
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Figure  2.6  Module  6,  Data  Generation. 


crucial  point,  the  analysts  must  decide  if  their  questions  can  be  answered  by  their 
analysis.  Credibility  and  reliability  are  major  concerns  to  the  decisionmakers. 
| Ref.  1:  pp.  21.22| 


C.  COURSES  OF  ACTION 


The  results  of  the  MCFS  iteration  are  provided  to  the  decisionmaker.  Figure  2.S 
represents  the  actions  and  results  of  each  iteration  of  the  MCLS  modules.  At  least  two 
courses  of  action  are  then  available  to  the  decisionmaker  based  on  the  results.  1  he 
decisionmaker  may  directly  implement  the  results  of  the  MCFS  evaluation  or  he  may 
identify  the  need  for  further  study  or  require  another  iteration  of  the  MCI  S  analysis. 
The  decisionmaker  may  interact  with  the  MCFS  analysis  effort  to  further  guide  the 
analysts  by  identifying  errors  in  assumptions,  clarifying  the  bounding,  etc.  The  analysis 
could  be  modified  by  infusing  new  directions  or  objectives  based  upon  the  results  of 
previous  MCFS  modules  completed,  for  example,  the  bounding  of  the  C“  system  may 


generate  the  observation  that  significant  interfaces  are  outside  the  oneinallv  conceive J 


scope  of  the  study  and  require  a  return  to  the  problem  formulation  module. 
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Figure  2.7  Module  7,  Aggregation  of  Measures  and  Interpretation. 

D.  USES 

MCES  can  assist  in  the  areas  of  management  and  analysis.  MCES  assists  in 
the  systematic  specification  of  the  problem  by  focusing  on  identified  essential 
characteristics  of  the  C~  system.  It  permits  a  senior  analyst  to  conduct  a  C 
evaluation  effectively.  MCES  assists  the  analyst  in  forming  a  concise  conclusion  and 
provides  the  manager  with  supporting  data  for  decisionmaking.  1  he  II  I  N  Testbed  is 
a  good  example  of  a  evaluation  of  air  defense  and  will  be  illustrated  in  the  following 
chapters. 
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Figure  2.8  MCES  Modules  and  Output. 


III.  IFFN  TESTBED  DESCRIPTION 


A.  THE  IDENTIFICATION  PROBLEM 

The  Identification  Friend,  Foe  or  Neutral  (IFFN)  Testbed,  comprised  of  a  U.S. 
Army  and  Air  Force  Joint  Test  Force  at  Kirtland  Air  Force  Base,  is  investigating  ways 
to  enhance  the  identification  of  friendly,  neutral,  and  enemy  aircraft.  A  realistic 
scenario  used  by  the  IFFN  Joint  Test  Force  forecasts  that  in  future  air  battles  our 
tactical  air  defenses  will  be  faced  with  sophisticated  enemy  fighters  capable  of  engaging 
our  forces  with  beyond  visual  range  (BVR)  weapons  The  large  number  of  enemy 
aircraft  with  their  use  of  low  level  tactics,  high  speeds,  and  Electronic  Countermeasures 
(ECM)  challenges  our  air  defense  systems.  The  air  defense  system  must  be  able  to 
identify  and  characterize  the  enemy  attack  and  then  direct  sufficient  force  in  time  to 
neutralize  it.  [Ref.  9:  p.  47] 

Effective  performance  of  the  active  air  defense  mission  requires  a  capability  to 
correctly  identify  aircraft  in  a  timely  manner  in  order  to  facilitate  the  air  defense 
weapon  system's  ability  to  employ  its  weapons.  This  air  defense  process  is  done 
through  a  complex  arrangement  of  personnel,  equipment,  communications  facilities 
and  procedures  which  form  a  C^  system.  This  requirement  is  particularly  important  in 
the  European  theater  where  large  numbers  of  friendly,  neutral,  and  enemy  aircraft  will 
be  part  of  the  tactical  air  environment.  In  this  environment,  surface-to-air  and  air-to- 
air  weapon  systems  must  operate  in  conditions  beyond  ranges  where  positive  visual 
identification  can  not  be  performed.  The  problem  is  aggravated  when  modern 
electronic  warfare  (EW)  threats,  particularly  those  of  the  Warsaw  Pact  are  considered. 
Numerous  studies  have  revealed  that  current  electronic  identification  capabilities  have 
numerous  problems  including  being  too  slow,  poor  at  positively  identifying  enemies 
and  friends,  insufficient  range  capability,  and  being  subject  to  interference  and 
electronic  countermeasures  (ECM).  The  inability  of  air  defense  weapon  systems 
operators  to  accurately  and  rapidly  discriminate  friend,  foe.  or  neutral  aircraft  results  in 
the  ineffective  use  of  these  weapon  systems.  These  problems  have  stimulated  activity 
within  the  US  military  and  NATO  to  develop  an  effective  NATO  identification  system. 
The  IFFN  Testbed  is  a  partial  attempt  by  the  United  States  at  solving  this  air  defense 
problem.  [Ref.  3:  p.  1] 


The  function  of  acquiring,  correlating,  fusing,  and  disseminating  direct  and 

•>  _____ 

indirect  IFFN  information  is  an  important  part  of  air  defense  C“.  This  1FFN 
information  serves  as  a  basis  for  threat  evaluation  and  engagement  control.  It  is  also 
the  function  of  air  defense  command  and  control  to  provide  sufficient  identity 
information,  bearing,  and  range  to  the  allocated  weapon  system  so  that  the  weapon 
system  is  able  to  acquire  and  engage  a  target  with  a  high  degree  of  confidence. 
Acquisition  and  engagement  information  processed  in  a  timely  manner  will  permit 
effective  weapons  employment.  An  IFFN  system  should  also  provide  the  necessary 
information  to  allow  passage  of  friendly  and  neutral  targets. 

B.  THE  AIR  DEFENSE  PROCESS 

Identification  is  an  integral  step  in  the  air  defense  process  and  begins  with 
tasking  and  disposition  of  assets  for  air  surveillance.  The  process  continues  through 
detection  of  an  intruding  target  and  ends  with  a  decision  to  engage  the  target  with  an 
air  defense  weapons  system.  This  process  is  characterized  by  complex  relationships 
between  air  surveillance,  C^.  and  weapons  systems.  The  targets  will  be  identified  as 
friendly,  neutral,  hostile,  or  unknown.  The  hostile  targets  will  be  given  different 
priorities  for  engagement  or  may  be  engaged  by  other  air  defense  assets.  Air  defense 
C~  must  have  sufficient  identity  information  to  evaluate  the  intent  of  hostile  targets 
and  assess  the  threat  level  posed  by  those  individual  targets.  Air  defense  must  then 
prioritize  those  targets  for  engagement,  allocate  the  targets  for  engagement  by  specific 
weapons  systems,  and  aid  a  weapon  system  in  acquiring  that  target  without 
endangering  other  targets.  This  must  be  done  in  a  constantly  changing  air 
environment  where  identity  determination  is  a  dynamic  process  involving  people, 
hardware,  and  software. 

The  classical  sequence  of  air  defense  is  detect,  identify,  engage,  and  destroy.  This 
classical  sequence  used  by  the  IFFN  Testbed  is  a  simplification  built  from  a  number  of 
decisions  and  functions  performed  separately  or  mutually  by  elements  and  air 
defense  weapon  systems.  Most  of  these  functions  and  decisions  are  dependent  upon 
identification.  Complicating  this  simple  sequence  is  the  fact  that  identification  is  both 
a  process  and  a  decision.  As  a  process,  identification  is  a  constant  gathering  and 
correlating  of  information  about  a  potential  target  from  all  sources  of  direct  and 
indirect  identification  information.  This  process  is  continuous  up  to  the  final 
disposition  of  a  target  by  the  air  defense  system.  As  part  of  the  identification  process. 


the  C*  system  must  correlate  information  from  all  sources  and  resolve  any  conflicts.  As 
a  decision,  an  identification  must  be  made  before  further  action  can  occur,  either 
actively  or  by  default.  This  decision  is  influenced  by  all  the  sensors  available  to  the  air 
defense  system,  by  the  background  intelligence  on  the  air  situation,  and  by  the 
operational  procedures  such  as  rules  of  engagement  and  weapons  control  status.  The 
decision-maker  must  either  make  or  delegate  the  identification  decision  prior  to  the 
engagement  decision.  [Ref.  3:  p.  SJ 

When  performing  the  classical  air  defense  sequence  within  an  integrated  air 
defense  system,  the  identification  decision  is  also  part  of  a  larger  process  which  is 
performed  by  both  C“  elements  and  air  defense  weapons  systems.  The  process 
becomes  more  complex  when  additional  sources  of  direct  and  indirect  information  are 
available  and  higher  levels  of  command  and  control  participate.  Individual  weapon 
systems  are  simultaneously  performing  detection,  tracking,  and  identification  as  part  of 
their  target  acquisition  function.  They  can  be  aided  in  performing  this  function  by  the 
command  and  control  system  as  it  exercises  its  function  of  engagement  control.  Wrhen 
operating  autonomously,  weapon  systems  are  limited  to  their  organic  detection, 
tracking,  and  identification  capability.  The  detection,  tracking,  and  identification  of 
the  entire  system  can  be  better  used  for  target  allocation  and  acquisition  when  the 
command  and  control  system  provides  engagement  control  in  a  centralized  mode  of 
operation.  Identification  is  a  major  factor  in  the  performance  of  weapon  systems  to 
defeat  the  enemy. 

C.  IFFN  JOINT  TESTBED  CONCEPT 

The  Department  of  Defense's  proposed  partial  solution  to  the  NATO  air  defense 
problem  was  the  development  of  the  IFFN  Joint  Testbed  to  gather  analytic  data  on 
the  problem  so  that  solutions  could  be  formulated.  The  testbed  will  assess  baseline  US 
capabilities  within  the  NATO  air  defense  C^  system  to  perform  the  IFFN  function, 
identify  deficiencies  in  the  performance  of  that  function,  and  propose  potential  near- 
term  procedural  and  equipment  modifications  for  further  testing.  One  issue  that  will  be 
addressed  by  the  IFFN  Testbed  is  the  indirect  information  process  and  how  its  use 
may  improve  the  performance  of  air  defense  systems  to  aid  C^  and  weapon  systems 
nodes.  A  testbed  approach  was  taken  so  that  a  number  of  different  C'-  strategies  could 
be  tested  without  having  to  actually  use  the  real  equipment  and  weapons.  The  testbed 
was  envisioned  to  simulate  as  close  as  possible  the  real  threat  and  the  L.S.  equipment 
and  procedures  used  in  NATO.  [Ref.  3:  pp.  1,2) 


The  level  of  complexity  of  the  air  defense  C“  system  is  enormous,  thus,  the  IITN 
Test  Force  expended  a  large  amount  of  elTort  in  understanding  general  C~  systems 
before  modeling  the  NATO  air  defense  system.  A  simulation  testbed  was  ultimately 
chosen  to  evaluate  the  alternative  air  defense  C~  systems.  The  IFFN  Test  Force  is 
attempting  to  determine  air  defense  identification  measures  of  performance  (MOPs) 
and  measures  of  etfectiveness  (MOEs)  that  will  lead  to  the  evaluation  of  the  utility  of 
the  different  configurations  of  the  air  defense  C^  architecture. 

1.  Baseline  Architecture 

The  IFFN  baseline  architecture  was  formulated  as  a  combination  of  hardware, 
software,  procedures,  and  doctrine  that  is  planned  to  exist  in  the  late  1980's.  The 
criteria  will  also  be  subjected  to  the  projected  Warsaw  Pact  1987-1990  threat  and  is 
consistent  with  Defense  Intelligence  Agency  estimates  of  enemy  capabilities  and  orders 
of  battle  for  that  period.  The  timeframe  chosen  was  a  compromise  between  possible 
near-term  benefits  and  results  which  will  have  long  range  applicability.  Certain 
modifications  that  will  be  fielded  in  1987  or  beyond  will  be  candidates  for  follow-on 
tests  using  the  IFFN  testbed.  [Ref.  3:  p.  3] 

The  testbed  geographical  area  of  interest  is  the  NATO  environment  in  which 
LS  Army  and  Air  Force  units  operate  jointly  and  under  the  control  of  associated 
elements  of  the  NATO  Air  Defense  Ground  Environment  (NADGE)  System.  The 
IFFN  testbed  will  focus  on  the  battle  management  area  of  a  representative  NATO 
Control  and  Reporting  Centers  (CRC)  located  in  the  Forth  Allied  Tactical  Air  Force 
(4ATAF)  area.  Representation  of  key  associated  NATO  command  and  control  nodes 
and  information  sources  is  required  in  the  IFFN  testbed.  Figure  3.1  depicts  the  key 
components  of  the  IFFN  Testbed.  [Ref  3:  pp.  3-6) 

a.  Command  and  Control  Units 

Command  and  control  units  are  those  representative  units  which  direct  or 
control  the  beyond  visual  range  (BVR)  weapon  systems  and  execute  the  active  air 
defense  mission.  The  specific  command  centers  that  perform  these  C^  functions  are: 
Sector  Operations  Center  (SOC),  Control  and  Reporting  Centers  (CRC),  Brigade  Fire 
Direction  Centers  (Bde  FDC),  and  Battalion  Fire  Direction  Centers  (Bn  FDC). 

b.  Information  Sources 

Sources  which  provide  information  for  identification,  target  allocation,  and 
target  acquisition  in  the  air  environment  to  the  C^  units  and  weapon  systems  are 
categorized  as  information  sources.  These  are:  NATO  Airborne  Early  Warning 
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Figure  3.1  IFFN  Air  Defence  Structure. 

systems  (NAEW).  Special  Information  (intelligence)  Systems  (Sh'>.  and  other 
information  sources  (i.e.,  (light  plans). 
c.  Weapon  Systems 

Weapon  systems  are  those  representative  BVR  weapon  s\stcm<  which  arc 
operated  by  US  forces.  For  the  IFFN  Testbed.  the  F-15.  HAWK,  and  l’A  I  RIOT  have 
been  selected  as  the  representative  BVR  weapon  systems.  Due  to  resource  constraints 
on  the  IFFN  Testbed,  only  these  three  weapon  systems  will  be  used. 

2.  Major  Operational  Issues 

The  testbed  will  generate  and  record  the  data  necessary  for  analysis  and 
recommendations  on  IFFN  issues  There  are  three  major  operational  issues  considered 
by  the  IFFN  Testbed  which  are  listed  below. 
a.  Issue  I 

"What  is  the  contribution  of  indirect  identification  information  to  the 
ability  of  US  air  defense  command  and  control  systems  operating  in  NATO  to 
correctly  identify  airborne  targets,  to  use  identification  in  performing  target  allocation, 
and  to  aid  subordinate  air  defense  weapon  systems  in  performing  target  acquisition1" 


Swirfw? 


[Ref.  10:  p.  2].  Indirect  ID  information  is  that  ID  information  that  is  not  direct 
electronic  IFF  returns.  Indirect  information  usually  refers  to  all  other  ID  information 
that  arrives  as  a  facility  such  as  flight  plans,  area  of  operations,  intelligence  reports, 
etc.  However,  direct  ID  information  once  passed  to  another  facility  also  becomes 
indirect  ID  information.  Satisfying  the  first  major  operational  issue  will  provide  a 
baseline  assessment  of  the  expected  identification  performance  of  a  representative 
NATO  air  defense  system  operating  in  the  4ATAF  area.  Studying  the  first  issue  will 
also  provide  a  fuller  understanding  of  the  relationship  between  the  identification 
performance  of  the  system  and  the  performance  of  the  overall  active  air  defense 
mission. 

b.  Issue  2 

"What  are  the  deficiencies  in  air  defense  system  use  (collection,  formation, 
dissemination,  and  use)  of  indirect  identification  information  for  which  solutions  are 
not  currently  planned?"  [Ref  10:  p.  2).  The  second  major  operational  issue  will 
identify  weaknesses  in  the  identification  process  and  allow  for  a  qualitative  comparison 
of  those  weaknesses  identified  during  testing.  Potential  corrective  actions  for  these 
identified  deficiencies  can  then  be  developed.  These  recommended  corrective  actions 
could  take  the  form  of  changes  in  doctrine,  procedures,  system  software, 
communications  connectivity,  or  addition  of  new  data  sources,  or  various  combinations 
of  these  solutions. 

c.  Issue  3 

"What  near-term  procedural  and  equipment  modifications  should  be 
recommended  to  overcome  deficiencies?"  [Ref.  10:  p.  2j.  This  issue  will  address 
productive  near-term  solutions  to  the  IFFN  problem. 

3.  Hybrid  Approach 

Two  major  options  were  considered  during  feasibility  studies  when  developing 
the  test  concept.  Field  exercises  and  computer-based  simulation  were  both  evaluated 
and  compared.  A  hybrid  approach  was  ultimately  selected  which  permits  the  use  of  the 
best  of  both  options.  The  concept  is  centered  around  live  operators  using  actual 
tactical  hardware  and  accepted  simulations  of  hardware  and  software  called  Live 
Participating  Units  (LPUs).  Real-time  computer  models  stimulate  the  LPUs  as  well  as 
represent  the  background  workload  for  these  units.  This  man-in-the-loop  simulation 
will  be  earned  out  through  all  the  tests.  Since  the  IFFN  Joint  Test  Force  developed  a 
hybrid  simulation  testbed  composed  of  simulated  participating  units,  there  was  no 
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requirement  for  live  operation  of  aircraft,  weapons,  radars,  nor  field  deployment  of 
weapons  and  command  and  control  systems.  To  implement  this  test  concept  a 
distributed  testbed  was  established  at  a  central  facility  at  Kirtland  Air  Force  Base,  New 
Mexico.  The  testbed  at  Kirtland  generates  and  distributes  the  tactical  scenario, 
controls  test  execution,  and  monitors  the  response  of  geographically  distributed  Patriot 
and  Hawk  LPt's  at  the  Army  Air  Defense  Center  at  Fort  Bliss,  Texas. 

A  realistic  scenario  environment  was  necessary  to  ensure  realistic  and  accurate 
results.  The  combination  of  a  few  high  Fidelity  LPUs  responding  to  simulated  and 
manned  units  was  determined  to  be  an  excellent  method  to  simulate  the 
interdependency  and  interaction  of  air  defense  units.  The  IFFN'  Test  Force  conducted 
a  substantial  testbed  certification  elTort  with  joint  service  participation  to  validate  and 
calibrate  testbed  performance  against  joint  and  combined  field  exercises  for  the  first 
test  series.  Further  credibility  and  validity  tests  will  be  made  after  each  test  series. 

4.  Models 

The  IFFN  models  that  were  developed  can  be  categorized  as  interactive  or 
noninteractive.  The  interactive  models  react  dynamically  to  perceived  changes  in  the 
air  battle  situation.  They  may  receive  inputs  such  as  data  link  messages  from  the  other 
models  or  LPUs  and  may  initiate  messages  in  response  to  this  input  or  their  own.  The 
output  of  these  models  is  dependent  on  the  specific  dynamics  of  the  air  battle.  The 
applications  of  the  interactive  models  for  the  IFFN  testbed  are:  sensor  models,  missile 
models,  and  dynamically  controlled  aircraft  models.  Examples  include  radar,  electronic 
IFF,  Patriot  SAMs,  and  fighter  and  attack  aircraft  platforms.  [Ref.  10:  p.  10] 

The  IFFN  noninteractive  models  do  not  react  to  the  air  battle  dynamics. 
They  are  less  complex  models  and  simply  generate  selected  messages  and  actions  at 
preprogrammed  times  according  to  a  script  prepared  prior  to  the  test.  The  models  are 
considered  suitable  for  simulating  those  facilities  that  do  not  dynamically  interact  with 
the  identification  process,  but  do  provide  orders,  procedures,  and  other  information  on 
a  one-way  basis  such  as  certain  higher  echelon  planning  facilities.  Noninteractive 
models  will  also  be  used  to  simulate  aircraft  following  programmed  flight  profiles  which 
are  not  automatically  reactive  to  the  air  battle  environment.  The  Sector  Operations 
Center  (SOC)  is  an  example  of  a  noninteractive  model  used  in  the  IFFN  Testbed 
simulation.  Examples  of  weapon  platforms  are  transport,  patrol,  and  tanker  aircraft. 
[Ref.  10:  p.  14) 


D.  TESTBED  PHASES 


In  order  to  minimize  technical  and  program  risks,  a  phased  testbed  acquisition 
was  adopted.  The  test  approach  is  based  on  seven  test  series.  The  series  will  consist 
of  weapons  systems,  command  and  control  systems,  and  associated  data  links.  One  of 
eight  planned  phased  simulations  has  been  completed.  The  following  is  a  description 
and  list  of  the  systems  involved.  [Ref.  10:  pp.  3.4] 

1.  Test  Series  1 

Series  1  tests  the  identification  performance  of  a  representative  US  \rmv 
Surface-to-Air  Missile  (SAM)  System  which  is  the  PATRIOT  fire  unit.  The  systems 
tested  are: 

1.  PATRIOT  Fire  Unit  (FU),  and 

2.  PATRIOT  Air  Defense  Information  Language  (PADIL). 

2.  Test  Series  2 

Series  2  adds  the  PATRIOT'S  first  echelon  of  command  and  control,  the 
Battalion  Fire  Direction  Center.  The  systems  tested  are: 
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1.  PATRIOT  FU, 

2.  PATRIOT  Battalion  Fire  Direction  Center  (Bn  FDC),  and 

3.  PADIL. 

3.  Test  Series  3 

Series  3  adds  the  next  level  of  C2,  the  Brigade  Fire  Direction  Center.  The 
systems  tested  are: 

1.  PATRIOT  FU, 

2.  PATRIOT  Bn  FDC. 

3.  PATRIOT  Brigade  Fire  Direction  Center  (Bde  FDC), 

4.  PADIL,  and  the 

5.  Army  Tactical  Data  Link- 1  (ATDIL  I). 

4.  Test  Series  4 

Series  4  tests  only  the  USAF's  fighter-intercepters,  the  F-15  .  The  system 
tested  is  the  F-15  "Eagle"  Intercepter. 

5.  Test  Series  5 

Series  5  adds  associated  USAF  C2  nodes  and  information  sources  to  the  F-15 
system.  The  systems  tested  are: 

1.  F-15, 

2.  USAF  Control  and  Reporting  Post  Message  Processing  Center  (CRP  MPC), 

3.  NATO  Airborne  Early  Warning  System  (NE-3A), 
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4.  Special  Information  System  (SIS), 

5.  Tactical  Digital  Information  Link.  -  A  (TADIL-A),  and 

6.  TADIL-B. 

6.  Test  Series  6 

Series  6  will  integrate  the  Army  systems  from  Series  1-3  with  the  L'SAF 
systems  from  Series  4  and  5.  This  will  now  be  a  joint  operations  test.  The  systems 
tested  are: 

1.  PATRIOT  FU, 

2.  PATRIOT  Bn  FDC, 

3.  PATRIOT  Bde  FDC, 

4.  F-15, 

5.  NE-3A, 

6.  CRP, 

7.  SIS, 

8.  TADIL-A, 

9.  TADIL-B, 

10.  PADIL, 

11.  ATDL-1,  and 

12.  NATO  Link-1. 

7.  Test  Series  7 

Series  7  will  add  a  CRC  to  form  the  total  system  to  be  tested.  The  systems  to 
be  tested  are: 

1.  PATRIOT  FU, 

2.  PATRIOT  Bn  FDC, 

3.  PATRIOT  Bde  FDC, 

4.  F-15, 

5.  NE-3A, 

6.  CRP, 

7.  SIS, 

8.  NATO  Control  and  Reporting  Center  (CRC), 

9.  TADIL-A, 

10.  TADIL-B, 

11.  PADIL. 

12.  ATDL-1,  and 

13.  NATO  Link- 1. 
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E.  IFFN  TEST  CELL  MATRIX 


The  simulation  will  be  conducted  using  a  controlled  variable  approach.  Different 
simulation  test  cells  are  used  in  which  some  variables  are  held  constant  while  others  are 
left  to  fluctuate  and  eventually  lead  to  the  determination  of  the  variables  impact  on 
the  C~  system.  The  basic  test  structure  considers  both  a  fully  integrated  air  defense 
system  and  several  subsets  of  the  system.  The  different  configurations  allow  different 
variables  to  be  isolated  to  establish  their  contribution  to  the  particular  air  defense  C“ 
system.  Each  trial  will  be  run  with  a  constant  measurement  volume  of  a  prescribed 
radius  with  only  predetermined  variables  changed.  The  measurement  volume  is  the 
area  covered  by  the  air  defense  unit  or  weapon  system.  The  test  cells  of  the  matrix  are 
used  to  generate  comparative  data  under  various  environmental  conditions.  Figure  3.2 
represents  the  basic  test  matrix  of  ten  test  cells  used  by  the  IFFN  testbed. 

Data  items  are  collected  from  collection  messages  that  are  generated  by  data 
events  during  the  simulation.  Data  events  are  events  that  take  place  during  the 
simulation  such  as  information  arrival  and  actions  performed  by  the  air  defense  C" 
system  nodes.  For  Test  Series  2,  there  are  fourteen  data  events  that  are  collected  from 
each  test  cell  simulation  to  be  used  in  the  calculation  of  the  MOEs  and  MOPs  with 
other  data  events  used  for  deficiency  analysis.  All  MOEs  and  MOPs  are  divided  into 
two  groups  of  probability  measures  and  distribution  measures.  These  measures  are  not 
necessarily  measures  of  the  variables  themselves  but  are  intended  to  be  measures  of  the 
results  of  the  variables  impact  on  the  system.  [Ref.  11:  p.  20] 

F.  MEASURES 

General  categories  of  measures  were  sought  to  derive  values  that  would 
eventually  lead  to  discrimination  among  the  different  systems.  The  simulation  can 
not  completely  characterize  the  performance  of  the  fully  deployed  air  defense  systems, 
so  absolute  conclusions  about  the  performance  of  the  air  defense  systems  are  nearly 
impossible.  With  this  shortcoming  in  mind,  measures  of  the  relative  change  in  the 
performance  effect  of  the  variable  under  varying  conditions  will  be  used  when  only- 
large  and  significant  differences  are  noted.  This  means  that  a  low  confidence  level  will 
be  used  when  analyzing  the  data.  Figure  3.3  depicts  the  general  approach  taken  by  the 
IFFN  Testbed  in  resolving  its  IFFN  issues. 


37 


M 

$ 

*;s» 


I 

\v» 


NOM  (Nominal) 

ACP's  (Airspace  Control  Procedures) 

Q&A  IFF  (Question  and  Answer  Electronic  IFF) 
FU  (Fire  Unit) 

HD  (Indirect  Identification) 

AUTO  ( Autonomous) 


Figure  3.2  If  FN  Basic  Test  Matrix. 


V.V’/I’.V,  V 


IFFN  APPROACH 

(Without  MCES) 

IFFN  ISSUES 

'U' 

TEST  OBJECTIVES 

o 

TEST  DESIGN 

Test  Matrix  of  Competing  Configurations 
Simulation  Testbed 

'U 

SPECIFICATION  OF  MEASURES 

Air  Defense  Functions  »  MOE 

Subfunctions  - ►  MOP 

DATA  GENERATION 


Figure  3.3  IFFN  Test  Design  Approach. 


1.  Test  Series  2  Objectives 

There  were  originally  six  objectives  formulated  from  the  three  issues  for  Test 
Series  2  that  eventually  led  to  the  formulation  of  the  IFFN  Testbed  measures.  These 
objectives  are  listed  below  without  the  more  detailed  sub-objectives  for  each  objective. 

1.  Objective  I  -  Assess  and  contrast  the  performance  of  the  PATRIOT  battalion 
under  centralized  and  decentralized  control. 

2.  Objective  2  -  Assess  the  impact  of  chansine  and  removine  Airspace  Control 
Procedures  (ACPs)  on  the  operational  performance  ot  the  centralized  and 
decentralized  battalion  and  autonomous  fire  unit. 

3.  Objective  3  -  Determine  the  value  and  impact  of  perfect  ID  and  communication 
system  performance  on  PATRIOT  battalion  performance. 

4.  Objective  4  -  Evaluate  the  impact  of  various  changes  in  direct  and  indirect  ID 
performance  and  interaction. 

5.  Objective  5  -  Assess  the  influence  of  the  fire  unit  ID  function  on  the  ablity  of 
the  PATRIOT  battalion  to  perform  its  functions. 

6.  Objective  6  -  Identifv  and  subjectivelv  evaluate  any  PATRIOT  operational 
deficiencies  noted  during  testing.  [Ref.  II:  pp.  2. 1,2.2] 
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2.  Measures  of  Effectiveness  (MOE) 

There  are  three  primary  MOE  areas  formulated  for  all  the  test  series.  The 
FFFN  Testbed  originally  identified  three  functions  that  were  performed  in  the  air 
defense  process:  identification,  target  allocation,  and  target  acquisition.  [Ref.  3:  pp. 

r.is| 

a.  MOEs  for  the  Identification  Function 

These  MOEs  will  describe  how  well  the  weapons  and  systems  are  able 
to  identify  or  recognize  airborne  objects  and  assign  them  to  appropriate  identification 
categories. 

b.  MOEs  for  the  Target  Allocation  Function 

These  MOEs  will  relate  identification  information  used  by  systems  to 
allocate  air  defense  weapons  against  hostile  aircraft  and  prevention  of  misallocation  of 
weapons  against  friendly  aircraft. 

c.  MOEs  for  the  Target  Acquisition  Function 

These  MOEs  provide  the  measures  relating  indirect  identification 
information  to  the  weapons  systems.  Target  acquisition  provided  by  the  command  and 
control  structure  to  the  weapons  systems  is  a  part  of  these  MOEs. 

d.  MOE  List 

The  MOEs  for  Test  Series  2  that  measure  the  needed  information  for  the 
C"  evaluation  issues  are  listed  in  Table  2  [Ref.  11:  p.  D25).  The  letter  "P"  identifies 
probability  measures  while  the  letter  R"  signifies  range  distributions  and  the  letter  T" 
represents  time  distributions. 

3.  Measures  of  Performance  (MOP) 

Measures  of  Performance  (MOP)  for  the  IFFN  Testbed  were  submeasures  of 
the  air  defense  functions  and  therefore  subsets  of  the  MOEs.  An  example  is  the 
probability  that  a  passed  ID  is  correct  which  is  a  submeasure  of  MOE  3,  probability  of 
identification  of  an  aircraft.  The  MOPs  for  Test  Series  2  that  were  determined  to 
measure  the  needed  quantities  or  qualities  to  resolve  the  six  stated  objectives  are  listed 
in  Table  3  [Ref.  11:  pp.  D23.D24]  with  corresponding  MOE  number  references. 

4.  Measures  of  Force  Effectiveness  (MOFE) 

Measures  of  Force  Effectiveness  are  global  measures  that  determine  how 
effectively  the  mission  is  accomplished.  Target  hits  and  damage  assessments  were  not 
modeled  in  this  testbed  so  MOFEs  could  not  be  used. 


TABLE  2 

MOE  DEFINITIONS 


MOE# 

MOE 

MOE  DEFINITION 

1. 

P(D/V) 

Probability  of  detecting  an  aircraft 
given  that  it  has  entered  a  system 
measurement  volume. 

2. 

P(  T/D) 

Probability  of  tracking  an  aircraft, 
given  that  it  has  entered  a  system  s 
measurement  volume  and  has  been  detected. 

3. 

P( I/T) 

Probability  of  identification  of  an 
aircraft,  given  that  it  has  entered 
a  system  s  measurement  volume  and  has 
been  tracked. 

4. 

P( A/I ) 

Probability  of  allocation  of  an 
aircraft,  given  that  it  has  entered 
a  system  s  measurement  volume  and  has 
been  identified. 

5. 

P(E/A) 

Probability  of  engagement  of  an 
aircraft,  given  that  it  has  entered 
a  system  s  measurement  volume  and 
has  been  allocated. 

6. 

P(E/V) 

Probability  of  engaging  an  aircraft. 

7. 

P(E/F) 

Probability  of  engaging  a  friedly 
aircraft. 

8. 

P  (  E/N ) 

Probability  of  engaging  a  neutral 
aircraft. 

9. 

P( E/H) 

Probability  of  engaging  a  hostile 
aircraft. 

10. 

T( Tot) 

Distribution  of  times  elapsed  between 
detection  and  engagement. 

11. 

P( EBO) 

Probability  of  engaging  a  hostile 
aircraft  before  it  fires 
a  missile  or  drops  ordnance. 

12. 

R(  FE) 

Distribution  of  ranges  from  aircraft  to 
FSCL  at  time  of  engagement. 
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TABLE  3 

MOP  DEFINITIONS 


MOE 

ref# 

MOP 

MOP  DEFINITION 

1) 

R(D) 

Distribution  of  ranges  from  aircraft 
to  detection  unit  at  time  of  detection. 

1) 

R(  FD) 

Distribution  of  ranges  from  aircraft  to 
FSCL  at  time  of  detection. 

2) 

R(T) 

Distribution  of  ranges  from  aircraft  to 
tracking  unit  at  the  time  the  track  was 
established. 

2) 

R(  FT) 

Distribution  of  ranges  from  aircraft  to 
FSCL  at  time  of  tracking. 

2) 

T(DT) 

Distribution  of  times  elapsed  between 
detection  and  tracking. 

3) 

P( IX/YI ) 

Probabilities  of  identifying  an  aircraft 
as  category  X  (friend,  neutral,  or 
hostile),  given  that  its  true  identity 
is  Y  (friend,  neutral,  or  hostile)  ana 
that  it  has  been  detected. 

3) 

R(  I) 

Distribution  of  ranges  from  aircraft  to 
identifying  unit  at  time  of  ID. 

3) 

R(  FI ) 

Distribution  of  ranges  from  aircraft  to 
FSCL  at  time  of  ID. 

3) 

T(TI) 

Distribution  of  times  elapsed  between 
tracking  and  ID. 

3) 

P( Pass ) 

Probability  that  a  passed  ID  is  correct. 

3) 

P( Res) 

Probability  that  an  ID  conflict  is 
resolved  while  the  aircraft  is  still  in 
the  weapon  system  s  measurement  volume. 

3) 

T( Trans) 

Distribution  of  times  elapsed  between 
receipt  and  retransmission  of  ID 
information  by  a  C2  node. 

3) 

P(  Amp) 

Probability  that  an  ID  includes  track 
amplification  information. 

4) 

P(  A/YI ) 

Probabilities  of  allocating  an  aircraft, 
given  that  its  true  identity  is  Y 
(friend,  neutral,  or  hostile)  and  tha^ 
it  has  been  identified. 

4) 

R(A) 

Distribution  of  ranges  from  aircraft  to 
allocated  unit  at  time  of  allocation. 

4) 

R(  FA) 

Distribution  of  ranges  from  aircraft  to 
FSCL  at  time  of  allocation. 
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TABLE  3 

MOP  DEFINITIONS  (CONT’D.  ) 

4)  T( IA)  Distribution  of  times  elapsed  between  ID 

and  allocation. 

5)  P(E/YA)  Probabilities  of  engaging  an  aircraft, 

given  that  its  true  identity  is  Y 
(friend,  neutral,  or  hostile)  and 
that  it  has  been  allocated. 

5)  R{E)  Distribution  of  ranges  from  aircraft  to 

engaging  unit  at  time  of  engagement. 

5)  T(AE)  Distribution  of  times  elapsed  between 

allocating  and  engagement. 


G.  ANALYSIS  OF  DATA 

1 .  Exploratory  Data  Screening 

The  test  data  results  are  first  examined  and  screened  for  outliers  and  to  verify 
underlying  assumptions  such  as  normality,  independence,  constant  variances,  and  zero 
mean.  Various  methods  are  used  to  statistically  check  the  data.  Data  screening 
involves  a  number  of  methods  listed  below.  [Ref.  11:  pp.  D37-D42] 

a.  Box  and  line  plots 

Box  and  line  plots  are  used  for  time  and  range  distributions.  The  box  or 
line  plot  allows  pictorial  presentation  of  a  set  of  distribution  measures  for  a  set  of  trials 
or  number  of  test  cells. 

b.  Frequency  distributions  ( Histograms ) 

Histograms  are  used  also  for  time  and  range  distributions.  These  show 
visual  evidence  of  normality  as  well  as  extreme  outlying  values. 

c.  Scatterplots 

Scatterplots  are  used  for  time  versus  range  plots  distributions.  Bivariate 
scatterplots  provide  a  visualization  of  the  relationship  between  two  continuous 
variables. 

d.  Normal  probability  plots 

Probability  plots  are  used  to  determine  probability  types  and  fits.  Normal 
probability  plots  provide  visual  evidence  of  the  difference  between  a  given  distribution 
and  a  Gaussian  distribution. 


2.  Data  Analysis 

The  data  analysis  that  follows  screening  is  listed  below. 

a .  Paired  T  Tests 

Means  are  tested  using  this  test.  The  assumptions  concerning  the 
underlying  populations  are  that  they  are  independent  samples  and  the  variances  are  the 
same.  If  they  are  not.  then  F  tests  are  used. 

b.  Analysis  of  Variance  (ANOVA) 

ANOVA  allows  inferences  to  be  formulated  about  differences  in  "treatment 
effects"  brought  about  by  the  test  variables  which  are  controlled  from  cell  to  cell. 
Inferences  are  made  by  estimating  how  much  of  the  variability  in  test  data  is  explained 
by  the  effect  of  the  test  variables  and  how  much  is  due  to  random  error. 

c.  Hypothesis  Testing 

The  ANOVA  provides  a  basis  for  the  formal  hypothesis  test  that  the  trial 
cell  means  or  specific  subgroup  means  are  all  equal. 

d.  Contingency  Table  Analysis 

This  analysis  is  sometimes  called  row  by  column  (RC)  table  analysis.  This 
test  is  used  when  more  than  two  outcomes  are  possible,  a  frequent  occurrence  among 
the  test  cells.  Each  2x2  contingency  table  analysis  will  be  performed  for  comparing 
probability  measures  from  cell  to  cell  on  as  many  of  the  measures  as  practicable. 

e.  Regression 

Regression  analysis  determines  the  statistical  relationship  between  one  or 
more  independent  variables  and  a  dependent  variable.  Curve  fitting  is  accomplished  by 
regression  of  the  dependent  variable  on  the  independent  variable. 

/.  Correlative  analysis 

The  relationship  between  the  independent  and  dependent  variables  or 
causality  is  determined  by  correlation  analysis.  This  analysis  determines  what 
proportion  of  the  variation  of  the  dependent  variables  can  be  attributed  to  the 
relationship  with  the  independent  variable. 

g.  Standard  Normal  Theory  Approximations 

These  series  of  tests  are  used  to  determine  what  type  of  probability 
distribution  exists  and  how  good  that  data  fits  that  distribution. 

h.  Deficiency  analysis 

After  the  data  has  been  checked  and  analyzed,  the  causes  of  the  differences 
in  the  configurations  are  proposed  and  examined.  The  objective  is  to  find  the 


underlying  cause  or  reason  behind  the  differences.  This  could  be  helpful  to 
decisionmakers  in  allocating  limited  resources  to  different  C~  configurations. 
(Ref.  11:  pp.  Dll -D29J 

H.  IFFN  PROGRESS  SUMMARY 

It  is  evident  that  the  IFFN  Testbed  has  made  progress  in  its  attempt  to  evaluate 
the  NATO  air  defense  system.  The  IFFN  air  defense  problem  is  definitely  complex 
and  the  IFFN  Testbed  has  understandably  committed  large  amounts  of  resources  to 
the  problem.  The  Testbed  is  a  good  concept  for  an  experimental  design  to  test 
competing  C“  systems  since  all  of  the  alternative  systems  can  not  be  tested  using  actual 
equipment  due  to  resource  constraints  or  present  configuration  limitations.  The 
IFFN  Test  Force  has  been  careful  to  insure  that  the  testbed  is  credible.  Only  Test 
Series  1  has  been  completed  with  Test  Series  2  to  begin  in  March  1987. 
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IV.  PROPOSED  MCES  APPLICATION  TO  THE  IFFN  TESTBED 


A.  INTRODUCTION 

Primary  research  into  the  application  of  MCES  as  an  evaluation  tool  for  the 
IFFN  Testbed  was  undertaken  by  Major  Patrick  Gandee,  U  S.  Air  Force,  when  he  was 
a  Naval  Postgraduate  School  student.  Two  Military  Operational  Research  Society 
(MORS)  teams  also  contributed  to  the  proposed  MCES  application  during  two  MORS 
conferences.  The  bulk  of  Major  Gandee's  and  the  MORS  teams'  work  with  the  IFFN 
Testbed  is  included  in  Major  Gandee's  thesis  [Ref.  12]  and  later  refinements  by  Major 
Gandee  as  a  staff  member  of  the  Joint  Chiefs  of  Staff  Command  and  Control  Systems 
Directorate  [Ref.  9:  pp.  49-58].  Dr.  Ricki  Sweet,  who  was  Major  Gandee's  thesis 
advisor,  provided  the  then  current  MCES  methodology  guidance.  This  application 
study  was  also  supported  by  the  Naval  Postgraduate  School  and  the  Office  of  the  Joint 
Chiefs  of  Staff.  Major  Gandee  received  assistance  from  IFFN  Testbed  personnel 
including  Colonel  Dave  Archino,  Director,  and  Major  Mike  Grey,  Chief,  Operational 
Analysis  Section.  These  IFFN  Testbed  personnel  also  participated  in  the  MCES 
application  to  the  IFFN  Testbed.  The  following  application  is  mainly  taken  from 
Major  Gandee's  thesis,  notes,  and  briefings  along  with  Major  Grey's  notes  and 
conversations  plus  research  of  IFFN  Test  Force  test  plans. 

B.  PROBLEM  FORMULATION 

Utilizing  the  first  MCES  module,  the  MORS  team  and  Major  Gandee 
reformulated  the  IFFN  problem  which  was  a  subset  of  the  air  defense  C^  process 
problem.  Within  air  defense  C^,  the  emphasis  was  on  allocating  multiple  hostile 
targets  to  weapons  systems  for  engagement.  Major  Gandee  understood  that  this  air 
defense  C^  process  description  should  consist  of  a  complete  set  of  battle  management 
functions  which  were  needed  to  direct  the  weapon  systems  to  perform  the  air  defense 
mission.  Other  issues  considered  by  Major  Gandee  in  the  first  module  were  the 
different  evaluation  levels  and  analysis  objectives.  Issues  such  as  procedural  control 
and  centralized  and  decentralized  control  were  also  researched  and  reviewed.  Figure 
4.1  lists  the  major  actions  taken  by  Major  Gandee  in  applying  MCES  to  the  IFFN 
Testbed. 


Figure  4.1  1FFN  Application  of  VICES  Module  1. 

The  IFFN  Testbed  had  focused  its  analysis  on  specific  concerns  of  Army,  Air 
Force.  NATO,  and  DOD  decision  makers  regarding  the  role  of  identification  as  it 
contributes  to  the  effectiveness  of  the  air  defense  process.  The  major  concern 
expressed  by  the  IFFN  Joint  Testbed  was  the  determination  of  how  the  programmed 
C“  system  and  weapon  systems  would  operate  together.  The  IFFN  mission  and  its 
environment  (friends,  foe.  neutral,  weather)  had  already  been  specified  prior  to  Major 
Gandee's  MCES  application.  The  friendly  weapon  systems  were  limited  to  SAMs  and 
fighters  with  beyond  visual  range  (BVR)  munitions.  A  conventional  threat  scenario 
was  already  chosen  by  the  IFFN  Test  Force  so  that  stress  on  the  CN  system  could  be 
affected  by  varying  traffic  volume,  ECM  jamming  to  radars,  communication  jamming 
and  varying  weather  conditions.  (Ref.  1:  p.  65] 

The  final  step  addressed  by  Major  Gandee  in  this  module  was  the  analysis 

y 

objective.  The  overall  air  defense  C  analysis  objective  was  reformulated  by  the 
January  1985  MORS  workshop  from  IFFN  issues.  The  analysis  objective  (Ref.  2:  p.  1] 
was  to  determine: 


How  effective  is  the  air  defense  O  svstem  in  the  central  region  in  Europe  in 
providing  decisionmakers  the  means  to  assess  and  employ  air  defense  assets  to 
meet  overall  mission  objectives? 


Major  Gandee  realized  that  identification  can  be  affected  by  the  presence  of  a  physical 
entity  or  an  asset  like  the  airborne  command  post  or  by  procedures  such  as  those  used 
for  passing  identification  information.  The  IFFN  analysis  objective  was  eventually 
expanded  by  Major  Gandee  to  determine: 

1.  How  effective  is  the  C2  process  when  the  C2  structure  and  its  attendant 
changes  in  tactics  and  procedures  is  varied. 

2.  How  effective  is  the  C*  process  when  physical  entities  are  added  or  lost. 
[Ref.  2:  p.  3] 

The  details  of  formulating  the  analysis  objective  involved  interaction  between  the 
decisionmakers,  operational  users  and  analysts.  [Ref.  12:  pp.  21-23] 

C.  C2  SYSTEM  BOUNDING 

In  the  next  module,  the  bounding  of  the  C2  system  of  interest  was  confirmed  by 
Major  Gandee  from  previous  IFFN  efforts.  Figure  4.2  lists  the  condensed  results  of 
implementing  Module  2  for  the  IFFN'  Testbed.  Physical  entities  were  identified  and 
bounded  and  Figure  4.2  depicts  the  successful  application  of  the  "onion  skin"  idea. 
Alternative  organizational  structures  were  determined  and  hierarchal  charts  formulated. 
Again,  much  of  this  had  been  accomplished  earlier  at  the  IFFN  Testbed  but  not  by 
using  specific  MCES  methodology.  The  application  of  this  module  confirmed  that  the 
IFFN  C2  bounding  was  sound.  [Ref.  12:  pp.  21-26] 


D.  C2  PROCESS  DEFINITION 

1.  Air  Defense  C2  Process  Functions 

In  this  module,  Major  Gandee  defined  the  C2  process  functions  of  the 
distributed  C2  air  defense  system.  The  air  defense  C2  process  functions  were 
determined  to  be:  detect  (D),  track  (T),  identify  (ID),  assess  threat  (TA),  assign 
weapon  (WA),  allocate  weapon  (AW),  and  weapons  monitor  and  control  (C).  Figure 
4.3  depicts  the  air  defense  C2  process.  These  air  defense  C2  process  functions  were 
mapped  to  the  modified  Lawson's  C2  process  model  to  ensure  that  all  C2  functions 
had  been  considered  and  Figure  4.4  depicts  that  translation.  [Pef.  12:  pp.  32-37] 

These  air  defense  functions  represented  what  the  air  defense  C2  system  and 
weapon  system  are  required  to  accomplish  together  to  perform  the  mission.  For  the 
IFFN  Testbed  application,  a  process  function  was  added  to  Lawson  s  generic  C2  loop 
and  the  plan  function  was  eliminated.  Lawson's  plan  function  did  not  correspond  to  a 
real-time  activity  during  the  IFFN  execution  phase,  however,  non-real  time  plans  such 
as  airspace  control  procedures  (ACPs)  and  rules  of  engagement  (ROE)  are  a  part  of 
weapons  assignment,  allocation,  and  control. 
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Figure  4.4  Mapping  of  Air  Defense  Cz  Functions. 


2.  Distributed  C"  Process  Interface 

Since  the  IFFN  Testbed  was  dealing  with  a  distributed  C~  system,  the 
determination  of  C*  process  function  boundaries  was  sometimes  complex.  Major 
Gandee  discovered  that  in  a  distributed  command  and  control  system  there  are  three 
distinct  processes  that  will  affect  the  overall  performance  of  the  system;  intelligence, 
crosstell  (coordination),  and  execution  level  processes.  [Ref.  12:  pp.  38.39J 
a.  XT  ELL  Process 

A  separate  Crosstell  (XTELL)  process  provides  a  way  to  share  target 
information  for  the  purpose  of  improving  the  overall  picture  of  the  environment  and 
improving  the  accuracy  of  information.  This  is  especially  important  for  identification 
(ID)  information  in  the  air  defense  application  where  command  centers  are 
geographically  dispersed.  Individual  command  centers  may  develop  definitive  ID 
information  which  can  be  used  by  other  command  centers  who  have  a  tactical 
advantage  or  resources  to  engage  the  target.  The  XTELL  process  is  accomplished 
through  three  functions  of  Crosstell  (XTELL),  Track.  Correlation  (TC).  and  ID 
Conflict  Resolution  (IDC).  The  XTELL  process  is  represented  in  Figure  4.5  with  its 
C~  process  interface. 

The  XTELL  function  of  the  XTELL  process  is  the  transfer  and  receipt  of 
information  via  data  link  with  some  rules  or  filters.  These  rules  specify  where 
information  is  to  be  sent  and  what  information  will  be  received.  The  Track  Correlation 
(TC)  function  resolves  location  and  track  numbering  disagreements  in  the  C2  system. 
The  ID  Conflict  Resolution  (IDC)  function  resolves  conflicts  that  may  arise  in  the 
identification  process  between  different  C^  nodes.  At  some  nodes,  this  IDC  function  is 
a  fusion  process  while  at  other  nodes  it  is  a  decision  process. 

Figure  4.6  represents  the  XTELL  process  in  a  lateral  relationship.  This 
lateral  relationship  represents  adjoining  units  of  the  same  level  passing  coordinating 
information  between  them.  This  information  is  then  fused  and  correlated.  A  vertical 
or  hierarchal  XTELL  relationship  can  also  be  present  in  distributed  C^  systems.  The 
vertical  XTELL  process  is  similar  to  the  lateral  relationship  except  that  now  the 
coordinating  information  flows  between  the  hierarchical  related  units.  The  fusion  and 
correlation  of  the  identity  and  track  information  may  be  different  than  that  in  the 
lateral  relationship  since  the  higher  level  unit  will  usually  have  more  voice  in  resolving 
conflicts  of  information.  The  alternative  C^  systems  have  various  configurations  of 
these  types  of  XTELL  relations  making  some  the  configurations  quite  complex. 
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Figure  4.5  XTELL  Process  Functions  and  C“  Process  Interface. 
b.  IXTELL  Process 

An  Intelligence  { IXTELL)  process  aids  decisionmakers  throughout  the  C“ 
system  in  forming  perceptions  of  enemy  capabilities  and  intentions.  The  IXTELL 
process  is  accomplished  through  four  functions  of  Sense  (S),  Process  (P).  Intelligence 
Correlation  (IC),  and  Assess  (A).  [Ref.  12:  pp.  39-42] 

The  function  which  collects  data  necessary  to  describe  and  forecast  the 
environment  is  termed  the  Sense  (S)  function.  The  function  that  transforms  data  into 
information  about  the  enemy  forces'  disposition  and  actions  is  termed  the  Process  (P) 
function.  The  Intelligence  Correlation  (IC)  function  correlates  intelligence  information 
with  track  and  ID  information.  The  Assess  (A)  function  is  performed  when 
information  is  examined  and  patterns  uncovered  that  indicate  the  actions  or  intentions 
of  the  enemy.  The  Assess  function  is  also  performed  when  patterns  are  utilized  to 
forecast  possible  future  changes  in  environment.  Figure  4.7  graphically  depicts  the 
IXTELL  and  C*  process  along  with  the  XTELL  process. 
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Figure  4.6  Lateral  XTELL  Process. 
c.  Execution  Level  C~  Process 

The  INTELL  and  XTELL  processes  support  the  Command  and  Control 
Process.  The  C“  process  can  be  viewed  at  a  level  which  directly  controls  weapon 
systems  and  at  a  higher  echelon  which  coordinates  the  efTorts  of  C*-  processes  which 
direct  the  weapon  systems.  Since  the  IFFN  Testbed  simulates  the  NATO  air  defense 
system  which  is  geographically  distributed,  the  C“  process  included  a  netting  of  the 
separate  command  centers  through  the  XTELL  process.  The  INTELL  process  will 
also  be  interfaced  with  some  of  the  process  functions.  The  interfacing  of  the  XTELL, 
INTELL,  and  C^  processes  together  by  communication  links,  protocols,  operational 
procedures  determined  the  overall  architecture.  Figure  4.8  lists  the  major  actions 
completed  in  Module  3.  [Ref.  12:  pp.  44-46] 

E.  INTEGRATION  OF  SYSTEM  ELEMENTS  AND  FUNCTIONS 

Prior  to  developing  measures,  Gandee  felt  that  a  model  or  architecture  that 
described  the  system  was  definitely  needed.  When  Gandee  attempted  to  establish  an 
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Figure  4.7  INTELl.  XTELL.  and  C“  Process  Interface. 
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Figure  4.8  IFFN  Application  of  MCES  Module  3. 
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architecture  for  the  air  defense  C“  system,  he  found  that  he  needed  a  number  of 
actions  not  listed  in  the  then  current  MCES  methodology.  The  methodology  for 
interfacing  all  the  processes  into  an  architecture  was  not  covered  completely  in  the 
original  version  of  MCES.  The  developers  of  MCES  adopted  this  idea  of  the 
integration  of  system  elements  and  functions  into  an  architecture  to  support  the  C* 
evaluation. 

1.  Structures 

Information  flow  through  the  air  defense  C^  process  functions  was  used  by 
Gandee  to  derive  a  natural  hierarchical  relationship  between  the  individual  C" 
functions  in  the  form  of  an  information  flowchart.  Later  command  organizations  and 
equipment  and  communications  alignment  were  also  related  to  form  a  organizational 
structure  chart.  [Ref.  9:  p.  54] 

Major  Gandee  needed  a  methodology  that  documented  this  internal 
processing  and  described  how  the  information  is  input  to  and  output  from  the  function. 
There  are  a  number  of  methods  available  to  formulate  and  describe  the  internal 
processing  of  C^  functions.  In  this  module,  a  specific  software  design  technique.  Data 
Flow-Oriented  Design  [Ref.  7:  pp.  99-115]  was  used  to  integrate  the  system  elements 
and  functions  of  the  system.  Thus,  this  input  output  relationship  could  form  a 
description  of  the  internal  information  flow  between  separate  process  functions  as 
required  to  perform  the  mission.  The  end  result  was  a  "structure"  for  a  particular 
version  of  the  system.  The  MCES  definition  of  "structure"  states  that  structure 
identifies  the  arrangement  and  interrelationships  of  physical  entities,  procedures, 
protocols,  concepts  of  operation,  and  information  patterns.  [Ref.  12:  p.  48] 

a.  Data  Flow-Oriented  Design 

In  the  first  step,  each  process  function  was  examined  and  the  data 
flowing  through  the  function  defined.  A  graphic  representation  of  this  process  is 
termed  a  data  flow  diagram  (DFD)  and  they  describe  the  input,  output  relationships 
that  exist  between  the  functions.  Figure  4.9  depicts  a  data  flow  diagram  (DFD)  for 
an  "execution  level"  process  at  a  single  command  node.  The  DFD  s  were  also 
applied  to  interface  the  INTELL,  XTELL,  and  Force  processes  with  the  C“  process  in 
the  distributed  IFFN  system.  Major  Gandee  described  how  that  information  flow 
linked  those  separate  processes  into  an  architecture  of  the  complete  C~  or  combat 
system.  [Ref.  12:  pp.  47.48] 
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Figure  4.9  Air  Defense  Single  Node  Data  Flow  Diagram. 
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b.  Transform  Analysis 

In  the  next  step,  the  C~  process  as  a  whole  was  reviewed  and  a  transform 
analysis  performed  on  the  DFD  to  determine  the  process  or  transform  center. 
Using  this  flow  of  information  into  and  out  of  each  function,  a  transform  analysis  was 
conducted  to  determine  the  information  transforming  process  center  where  information 
(lowed  in  and  were  the  information  was  transformed  into  an  output  in  the  form  of 
control  information.  A  function  is  analogous  to  a  data  flow  transform.  An 
example  is  shown  in  Figure  4.9  which  depicts  the  Assess  Threat  function  as  the  C~ 
process  or  transform  center  of  the  basic  air  defense  process  where  the  main  perception 
is  formed.  Information  flows  in  to  formulate  this  perception  and  is  termed  the  afferent 
branch.  Information  flows  out  in  terms  of  decisions  based  on  this  perception  and 
constitutes  control  flow  and  is  termed  the  efferent  branch.  Structurally,  this  Assess 
Threat  function  subordinates  the  others  and  it  was  designated  as  the  process  center. 
A  structure  chart  was  derived  from  this  transform  analysis  which  shows  the  overall 
structural  relationships  between  each  process  function.  This  process  center  was 
then  used  to  establish  the  structural  hierarchy  between  process  functions.  This 
hierarchical  relationship  between  C2  process  functions  was  presented  as  a  structure 
chart.  All  these  functions  can  be  potentially  performed  in  a  single  node  or  be 
distributed  between  different  nodes. 

After  the  functional  structure  is  formed,  the  people  and  their  equipment 
can  then  be  matched  to  that  structure.  Major  Gandee's  gave  an  example  of  the  battle 
commander  performing  the  Assess  Threat  function  supported  by  the  identification 
officer.  The  commander  also  has  subordinate  weapon  assignment  officers  who 
implement  his  decisions  to  attack  the  most  important  targets.  Major  Gandee  found 
that  the  equipment  consoles  can  be  matched  to  this  structure  as  capabilities  to  assign 
targets  or  control  weapons  can  be  implemented  by  configuring  consoles  to  address 
their  output  in  accordance  with  the  specified  "structure".  [Ref.  12:  pp.  48-50] 

c.  Null  Process  Concept 

2 

Under  some  operational  concepts,  process  functions  can  be  distributed 
between  command  nodes  such  as  Brigade  and  Battalion  FDCs  or  between  command 
nodes  and  weapon  systems  such  as  the  CRC  and  fighter.  The  C2  process  functions  can 
be  divided.  Such  arrangements  are  often  temporary  and  unique  to  the  particular 
version  of  the  system.  Major  Gandee  developed  the  concept  of  a  null  process  to 
differentiate  between  the  process  functions  when  they  are  distributed.  For  example. 


the  Brigade  FDC  allocates  to  the  Battalion  FDC  and  the  Battalion  FDC  allocates  the 

■) 

weapon  system.  Only  one  C“  process  can  direct  a  weapon  system  although  its 
decisions  may  be  influenced  by  information  coming  from  other  processes.  Influence 
can  come  in  the  form  of  an  indirect  ID  or  priorities  from  a  higher  echelon.  Each 
command  node  can  potentially  perform  all  functions  to  direct  force  actions  in  the 
environment.  Figure  4.10  and  Figure  4.11  depict  the  distribution  of  and  force 
functions  between  a  Battalion  FDC  and  SAM  battery  for  two  differing  operational 
concepts  of  centralized  and  decentralized  control.  A  function  can  be  null  at  a  facility 
due  to  a  physical  limitation  such  as  the  null  Detect  function  at  the  Battalion  for  lack  of 
organic  radar  or  due  to  a  redistribution  of  decision  functions  to  reflect  a  different 
operational  doctrine. 

With  these  techniques,  the  system  architecture  was  changed  to  show 
relationships  between  "physical  entities",  and  "processes",  to  produce  a  "structure". 
This  structure  was  altered  to  reflect  different  operational  concepts  which  form  the 
different  versions  of  the  system.  There  should  be  a  different  structure  for  each 
alternative  system.  Figure  4.10  is  a  good  example  of  the  structure  of  a  battalion 
employing  centralized  control  of  its  battery  fire  units  and  is  different  than  the  structure 
in  Figure  4.11  of  a  Battalion  employing  decentralized  control  of  its  battery  fire  units. 
In  these  illustrations,  the  null  functions  are  not  enclosed  by  a  box.  (Ref.  12:  pp.  51-53] 
d.  Procedures 

Procedures  are  utilized  in  the  internal  processing  within  functions.  For 
instance,  some  IFFN  issues  deal  with  ID  value  of  air  space  control  procedures.  These 
rules  or  procedures  are  specified  externally  but  used  internally  within  the  ID  function 
to  determine  ID.  These  rules,  when  combined  with  other  sources  for  ID  into  some 
decision  loop  or  algorithm,  affect  the  internal  "structure"  of  the  ID  function.  If  these 
procedures  are  taken  away  then  the  decision  loop  (internal  structure)  is  changed. 
Major  Gandee  used  a  design  technique  which  provides  a  module  description  that 
explodes  each  function  to  define  the  internal  processing  and  the  coupling  to  other 
or  force  functions.  In  this  approach  the  functions  were  related  to  an  appropriate 
physical  entity  prior  to  determining  relevant  measures  of  performance  (MOP), 
measures  of  effectiveness  (MOE),  and  measures  of  force  effectiveness  (MOFE). 
[Ref.  9:  pp.  54,55] 


2.  Architecture 

The  MCES  defines  architecture'-  as  an  description  of  a  integrated  set  of 
systems  whose  physical  entities,  structure  and  functions  are  coherently  related.  The 
architecture  provides  a  representation  which  will  eventually  lead  to  the  ability  to 
measure  the  C“  system  response  and  the  effectiveness  of  directing  forces  to  accomplish 
the  mission.  The  integration  of  the  system  elements  of  man  and  machine  with  the 
process  functions  will  eventually  form  an  overall  architecture  that  can  be  used  by 
analysts  to  evaluate  the  C“  system.  The  final  step  is  to  formulate  a  overall  system 
architecture  that  will  incorporate  all  the  different  version  structures  internally.  The  last 
step  in  completing  the  architecture  was  to  identify  what  physical  entites  performed  the 
individual  process  functions  and  what  connectivity  linked  the  functions  together.  This 
established  a  single  architecture  represented  as  an  overall  structure  chart.  The  final 
form  of  the  architecture  includes  the  process  description  and  system  elements 
performing  the  processes  arranged  in  a  structural  framework.  Additional  modules  to 
the  structure  chart  provided  documentation  for  equipment,  personnel  requirements, 
and  connectivity  in  the  necessary  detail. 

The  general  architecture  will  remain  unchanged  but  the  structure  variations 
would  be  represented  by  the  different  version's  unique  information  flow.  An  air 
defense  example  was  illustrated  by  Major  Gandee  to  describe  how  structures  can  differ 
internally  in  a  system  architecture.  This  illustration  involved  air  defense  operators 
located  in  front  of  consoles.  Equipment  consoles  could  be  configured  in  various  ways 
to  aid  the  operator  in  performing  certain  functions  and  allow  the  output  to  be 
transfered  to  other  consoles.  The  operator  would  be  aided  in  his  ability  to  process 
information  and  communicate  it  through  a  machine  structure  that  parallels  an 
organizational  structure.  The  general  architecture  would  be  the  same  but  there 
would  be  unique  structures  utilizing  the  equipment  and  personnel  differently. 
[Ref.  12:  pp.  48-50] 

Major  Gandee  listed  three  advantages  for  utilizing  an  architectural 
representation  of  the  system.  Major  Gandee  found  that  the  process  can  be 
broken  down  into  separate  functions  and  appropriate  attributes  defined  more 
systematically  than  previous  brute  force  or  exhaustive  listing  methods.  For  example, 
the  major  Identify  function  attributes  were  relatively  easy  to  determine  as  accuracy, 
timeliness  and  completeness.  The  second  advantage  of  an  architectural  representation 
is  the  cabability  of  defining  where  a  measure  should  be  taken  to  measure  a  certain 


function.  Certain  operational  concepts  have  the  same  function  performed  at  different 
nodes  or  levels.  Therefore,  measurements  must  take  place  where  the  function  is  being 
conducted.  For  example,  the  Allocate  Weapons  (AW)  function  is  performed  at  the 
Battalion  level  in  the  Centralized  Control  mode.  Figure  4. in  .  but  is  performed  at  the 
Fire  Unit  level.  Figure  4.11  ,  in  the  Decentralized  Control  mode.  The  Battalion  just 
monitors  the  Allocate  Weapons  function  activities  in  the  Decentralized  Control  mode. 
II'  an  accurate  architecture  depicts  the  actual  operations  of  the  C“  system,  these 
relationships  are  clearly  delineated.  A  third  very  important  reason  for  architectural 
representation  is  its  capaoility  to  graphically  depict  the  C“  system  and  weapon  systems 
and  highlight  appropriate  operational  issues.  [Ref.  9:  pp.  54-56] 

Major  Gandee  s  work  did  result  in  the  addition  of  more  objectives  and 
ultimately  additional  measures  to  Test  Series  2  as  new  relationships  were  uncovered. 
Figure  4.12  displays  the  major  results  for  the  implementation  of  Module  4  to  the  IFFN 
Testbed. 
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rigure  4.12  IFFN  Application  of  MCES  Module  4. 


F.  SPECIFICATIONS  OF  MEASURES 

Major  Gandee  used  a  different  approach  than  the  IFFN  Testbed  when 
determining  the  measures  needed  to  evaluate  the  competing  versions  of  the  air  defense 
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C"  systems.  Instead  of  moving  from  issues  to  objectives  to  measures  to  answer  tho<e 
objectives.  Major  Gandee  examined  the  information  flow  and  bounded  system  elements 
to  determine  which  measures  were  needed.  Figure  4.13  graphically  depicts  Major 
Gandee  s  approach  for  determining  measures  for  the  air  defense  C“  system.  In  this 
figure,  the  shaded  area  represents  the  functional  air  defense  system.  The  MOPs  are 
measures  of  each  function  within  the  C“  system.  Each  function  is  dependent  on  those 
functions  preceding  it,  so  the  MOPs  are  conditional  probability  measures  with 
additional  range  and  timeliness  measures  not  shown.  MOEs  measure  how  well  the  C" 
system  performed  all  its  functions  as  a  whole  and  is  measured  outside  of  the  C“  system. 

1.  MCES  Test  Series  2  Issues 

Major  Gandee  illustrated  some  of  the  issues  he  considered  concerning 
identification,  centralized  control,  and  network,  connectivity  for  Test  Series  2.  Focusing 
upon  such  issues  lead  to  the  differentiation  among  the  alternative  architectures. 

•  Issue  1:  Will  centralized  control  at  the  Battalion  manage  the  missile  resources 
better  bv  spreading  the  fire  power  more  evenly  over  subordinate  units  and  over 
time? 

•  Issue  2:  Under  what  traffic  volume  conditions  can  centralized  control  be 
handled  without  degradation? 

•  Issue  3:  If  the  data  links  (XTELL  process)  carrv  information  on  which  targets 
have  been  allocated  for  engagement,  can  SAM  batteries  operate  in  a  self¬ 
deconflicting  manner  conserving  missile  resources? 

•  Issue  4:  Given  decentralized  control  and  self-deconflictine  doctrine,  is  a  fullv 
connected  data  network  required  to  prevent  a  single  point  failure  due  to  the 
possible  destruction  of  a  Battalion  Firing  Unit? 

•  Issue  5:  Will  the  XTELL  network  supply  the  most  complete  ID  to  the  other 
SAM  batteries  when  their  ID  equipment  becomes  inoperable?  [Ref.  9:  pp. 
5T.5S] 

2.  MCES  Measures 

Major  Gandee's  first  four  issues  for  Test  Series  2  were  sensitive  to  structural 
changes.  Major  Gandee  suggested  possible  efficiency  and  coordination  measures  which 
would  reflect  the  probability  of  a  target  being  engaged  by  more  than  one  unit  due  to 
lack  of  coordination.  The  fifth  issue  of  identification  questioned  whether  a  network 
could  increase  individual  unit  identification  capabilities.  This  could  be  accomplished  by 
supplying  more  complete  ID  information  from  other  units  which  formulate  the  ID 
information.  Major  Gandee  suggested  that  an  ID  accuracy  measure  could  be  used  to 
compare  the  accuracy  of  an  ID  formulated  at  an  organic  unit  and  the  ID  information 
which  passes  over  the  network  to  other  units  as  a  system  ID.  Examples  of  generic 
MOEs  are:  timeliness,  accuracy,  survivability,  capacity,  and  percent  completion. 
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Other  possible  measures  suggested  by  Major  Gandee  were  fusing  measures  that 
measured  the  ability  of  units  to  accept  and  fuse  system  ID  information  with  its  own 
organic  ID  information  to  improve  the  ID  accuracy  in  time  to  use  it  effectively.  Figure 
4.14  represents  the  major  measures  recommended  by  Major  Gandee  during  the  MCES 
application  of  Module  5.  [Ref.  12:  pp.  71-76) 
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Figure  4.14  IFFN  Application  of  MCES  Module  5. 


G.  MCES  APPLICATION  SUMMARY 

Major  Gandee's  proposed  application  of  MCES  to  the  IFFN  basically  stopped  at 
Module  5,  Specification  of  Measures,  due  to  time  constraints  and  the  delay  of  Test 
Series  2  execution.  As  a  result,  the  MCES  data  generation  and  aggregation  and 
interpretation  modules  were  not  evaluated  for  Test  Series  2  by  Major  Gandee.  The 
MCES  methodology  provided  a  evaluation  methodology  to  assist  the  IFFN  Test  Force 
in  its  evaluation  of  the  air  defense  C^  problem.  The  MCES  approach  of  systematically 
outlining  the  physical  entities,  structure,  and  C^  process  functions  insured  that  all 
evaluation  areas  were  covered.  MCES  has  definitely  helped  in  highlighting  the 
functional  measures  that  have  been,  overlooked  in  previous  C*  evaluations.  The 


distributed  functions  are  required  to  characterize  the  coordination  and  intelligence 
sharing  of  distributed  C“  systems.  There  are  different  levels  of  these  distributed 
functions  and  the  C“  structures  can  become  very  complex.  MCES  s  distributed 
functions  did  assist  in  describing  these  complexities.  The  MCES  concept  of  using  a 
model  or  architecture  to  establish  a  baseline  with  alternative  structures  to  represent  the 
competing  C~  systems  was  very  useful  in  developing  measures  to  differentiate  between 
them.  Understanding  the  system  has  to  take  place  before  attempting  to  measure  its 
utility  and  to  uncover  which  variables  are  responsible.  The  MCES  approach  appeared 
to  be  detailed  and  complete  and  new  relationships  were  uncovered  by  Major  Gandee. 
MCES  provided  the  1FFN  C^  analysts  with  a  theoretical  framework  for  determining 
the  utility  of  a  C^  system.  The  MCES  application  did  assist  the  IFF\  Testbed. 


V.  ANALYSIS  OF  THE  IFFN  TESTBED  AND  MCES  APPROACHES 


A.  FEEDBACK  TO  THE  DECISION-MAKER 

Major  Grey,  Chief.  Operational  Analysis  section.  IFFX  Joint  Test  Force,  and  the 
IFFN  testbed  director.  Colonel  Dave  Archino.  participated  in  the  MORS  workshop 
and  later  incorporated  some  of  the  MCES  ideas  into  its  test  plans.  Major  Gandee 
visited  the  IFFN  Testbed,  conducted  his  research,  and  made  his  MCES  application  to 
the  IFFN  Testbed  with  the  full  assistance  of  Major  Grey.  Major  Gandee's  proposed 
application  of  MCES  to  the  IFFN  basically  stopped  at  Module  5,  Specification  of 
Measures,  due  to  time  constraints  and  the  delay  of  Test  Series  2  execution.  As  a 
result,  the  MCES  data  generation  and  aggregation  and  interpretation  modules  were  not 
evaluated  for  Test  Series  2  by  Major  Gandee.  Due  to  even  another  delay  in  the 
execution  of  Test  Series  2  until  March  1987,  the  final  results  were  not  evaluated  during 
the  conduct  of  this  thesis  as  was  originally  planned. 

Test  Series  1  was  planned  and  conducted  without  the  aid  of  MCES.  The  early- 
planning  stages  of  Test  Series  2  had  been  completed  before  the  MCES  application 
started.  However,  when  new  or  better  ideas  were  developed  in  the  MCES  application, 
some  of  these  ideas  were  added  to  the  Test  Series  2  test  plan.  A  strict  comparison  of 
the  IFFN  Testbed  produced  Test  Series  1  results  to  Test  Series  2  would  not  be  valid 
due  to  the  mixed  participation  in  Test  Series  2.  In  the  evaluation  of  Test  Series  2.  it 
was  sometimes  difficult  to  determine  exactly  what  part  of  the  test  plan  was  attributable 
to  MCES  or  the  IFFN  Testbed.  In  almost  all  cases,  MCES  at  least  confirmed  earlier 
IFFN  Testbed  planning.  In  some  cases,  MCES  provided  new  insights  that  were 
responsible  for  a  better  test  plan. 

B.  PROBLEM  FORMULATION 

Although  the  problem  formulation  was  completed  earlier  by  the  IFFN  Test 
Force,  MCES  was  used  to  verify  that  the  correct  steps  were  taken.  As  previously- 
noted,  the  analysis  objective  was  expanded  by  the  MCES  application.  The  problem 
formulation  revolved  around  the  air  defense  problem  and  not  around  how-  to  build  a 
credible  testbed  to  evaluate  competing  air  defense  C^  systems.  The  IFFN  Testbed 
itself  was  a  system  that  could  be  evaluated  just  as  the  IFFN  Testbed  was  trying  to 
evaluate  air  defense  C^  systems.  The  IFFN  Testbed  was  evaluated  as  part  of  its 
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certification  effort.  Measures  were  formulated  that  would  eventually  determine  if  the 
IFFN  Testbed  produced  valid  and  credible  results.  A  comparison  of  the  results  was 
conducted  between  the  values  of  the  measures  of  target  identification,  allocation,  and 
engagement  from  the  IFFN  Testbed  simulation  and  actual  Patriot  Iivefire  exercises. 
The  building  of  the  testbed  itself  was  a  background  problem  and  will  be  covered  in 
greater  detail  in  the  data  generation  discussion. 

C.  C2  SYSTEM  BOUNDING 

The  bounding  of  the  C2  system  of  interest  was  confirmed  by  Major  Gandee  from 
previous  IFFN  efforts.  Physical  entities  were  already  identified  and  bounded.  Much  of 
this  had  been  accomplished  earlier  at  the  IFFN  Testbed  but  not  by  using  the  specific 
MCES  methodology.  The  application  of  this  module  confirmed  that  the  IFFN  C2 
bounding  was  sound.  The  "onion  skin"  idea  was  a  useful  tool  and  was  subsequently 
used  by  the  IFFN  Testbed  to  graphically  display  their  bounding. 

D.  C2  PROCESS  DEFINITION 

The  IFFN  Testbed  originally  identified  three  functions  that  were  performed  in 
the  air  defense  process  and  those  were:  identification,  target  allocation,  and  target 
acquisition.  The  IFFN  Testbed  did  conduct  a  functional  analysis  of  the  air  defense 
process  though  not  in  as  much  detail  as  the  MCES  functional  analysis.  Major  Gandee 
and  the  MORS  team  defined  the  C2  process  functions  of  the  distributed  C2  air  defense 
system  as:  detect  (D),  track  (T),  identify  (ID),  assess  threat  (TA),  assign  weapon 
(WA).  allocate  weapon  (AW),  control  (C)  which  were  later  adopted  by  the  IFFN 
Testbed. 

E.  SYSTEM  ELEMENTS  AND  FUNCTIONS 

The  IFFN  Testbed  did  formulate  the  alternative  C2  systems  that  it  wanted  to 
test.  Organizational  and  equipment  charts  were  constructed  as  well  as  some 
information  (low  charts  by  the  IFFN  Test  Force.  The  IFFN  Testbed  baseline  criteria 
was  basically  a  baseline  architecture  from  which  planned  derivations  would  be  tested. 
In  this  manner  the  IFFN  Testbed  accomplished  an  integration  of  system  elements  and 
functions  but  in  less  detail  than  Major  Gandee's  application.  The  MCES  application 
by  Major  Gandee  was  more  thorough  with  his  information  flow  charts,  organizational 
charts,  and  structure  charts.  Alternative  organizational  structures  were  determined  and 
hierarchal  charts  formulated  by  both  the  IFFN  Testbed  and  MCES  approaches. 


however.  Major  Gandee  s  structures  were  more  detailed  and  complete.  Major  Gandee. 
through  his  use  of  null  functions  and  adding  or  deleting  physical  entities,  developed 
many  different  configurations  for  possible  testing.  The  alternative  configurations  for 
centralized  and  autonomous  (decentralized)  control  of  the  Patriot  Fire  Units  was  one 
of  Major  Gandee's  contributions.  This  concept  was  added  to  the  IFFN  Testbed's 
analysis. 

F.  SPECIFICATION  OF  MEASURES 

General  categories  of  measures  were  sought  by  the  IFFN  Testbed  to  derive 
values  that  would  eventually  lead  to  discriminating  among  the  competing  air  defense 
systems.  Early  in  the  development  of  the  IFFN  Testbed,  the  analysts  realized  that 
they  could  determine  differences  between  the  competing  C^  systems  without  precisely 
knowing  which  variables  were  responsible  for  the  difference.  As  the  development  of 
IFFN  Testbed  progressed,  the  analysts  knew  they  needed  to  determine  which  variables 
were  causing  the  difference. 

1.  Deriving  Measures 

When  and  where  does  the  analyst  derive  measures  in  a  C^  system?  There 
were  two  different  approaches  considered  by  the  IFFN  Testbed  as  they  derived 
measures  to  evaluate  competing  C^  systems.  The  MCES  approach  utilized  the  C“ 
architecture  and  unique  structures  to  derive  their  measures.  The  MCES  methodology- 
built  a  baseline  C^  architecture  with  alternative  structures  to  derive  where  and  when 
the  measures  should  be  determined.  The  IFFN  Testbed  approach  utilized  issues, 
objectives,  and  subobjectives  to  derive  their  measures  and  later  built  a  baseline 
architecture  to  determine  where  to  use  the  measures.  The  Institute  for  Defense 
Analysis  (IDA)  conducted  extensive  research  in  their  attempt  to  determine  measures 
for  use  by  the  IFFN  Testbed  [Ref.  13].  IDA  also  basically  used  a  functional 
decomposition  of  the  air  defense  C^  system  to  derive  its  measures.  Alphatec.  Inc. 
developed  a  petri  net  model  of  the  IFFN  air  defense  C"  system  and  then  identified 
measures  to  determine  the  characteristics  of  each  interconnection  in  the  net  (Ref.  14], 
This  massive  effort  resulted  in  over  200  measures  for  their  five  levels  of  the  system. 

There  were  originally  six  objectives  formulated  for  Test  Series  2  that  led  to  the 
IFFN  Testbed  measures.  A  new  list  of  objectives  was  formulated  after  the  MCES 
application  and  appeared  in  the  next  revised  version  of  the  Test  Series  2  test  plan  and 
is  listed  below.  Objectives  2.  6,  and  ”  were  added  to  Test  Series  2  alter  the  MCES 
application  uncovered  the  need  for  them. 


•  Objective  1  -  Assess  and  contrast  the  performance  of  the  PATRIOT  battalion 
under  centralized  and  decentralized  control. 

•  Objective  2  -  Evaluate  the  contribution  of  adding  (Patriot  Battalion  FDC)  to 
autonomous  Patriot  tire  unit  performance. 

•  Objective  3  -  Assess  the  impact  of  changing,  and  removing  Airspace  Control 
Procedures  (ACPs)  on  the  operational  performance  of  the  centralized  and 
decentralized  battalion  and  autonomous  lire  unit. 

•  Objective  4  -  Determine  the  value  and  impact  of  perfect  ID  and  communication 
system  performance  on  PATRIOT  battalion  performance. 

•  Objective  5  -  Evaluate  the  impact  of  various  changes  in  direct  and  indirect  ID 
performance  and  interaction. 

•  Objective  6  -  Evaluate  the  contribution  of  Question  and  Answer  (Q&A)  IFF 
devices  to  Patriot  Bn  and  FL  performance. 

•  Objective  7  -  Evaluate  the  abilitv  of  indirect  ID  information  to  compensate  for 
the  loss  of  the  direct  Q&A  IFF  device  at  a  Patriot  FU. 

•  Objective  8  -  Assess  the  influence  of  the  fire  unit  ID  function  on  the  ablitv  of 
the  PATRIOT  battalion  to  perform  its  functions. 

•  Objective  9  -  Identifv  and  subjectivelv  evaluate  anv  PATRIOT  operational 
deficiencies  noted  during  testing.  [Ref.  II:  pp.  2. 1,2.2]' 

2.  MOEs  versus  MOPs 

The  terminology  used  by  the  IFFN  Testbed  for  MOEs  and  MOPs  is  directly 
opposite  of  the  MCES  terminology.  MCES  states  that  MOEs  are  measured  outside  of 
the  C*  system  and  that  MOPs  are  measures  of  the  functions  within  the  C“  system. 
MCES  MOPs  are  used  for  the  system  measurements.  Examples  of  MCES  MOPs 
would  be  probability  of  detection  and  correct  identification.  Within  the  force 
boundary,  MCES  Measures  of  Effectiveness  (MOEs)  are  used  for  measuring  the 
functions  of  the  C^  system.  Examples  of  generic  MCES  MOEs  are:  timeliness, 
accuracy,  survivability,  capacity,  and  percent  completion.  MOFEs  are  used  for  the 
boundary  measurements  between  the  force  and  the  environment.  An  MCES  example 
might  be  the  number  of  enemy  aircraft  destroyed  prior  to  releasing  their  weapons. 

The  IFFN  Testbed  termed  the  functional  measures  as  MOEs  and  their 
corresponding  submeasures  as  MOPs.  The  IFFN  Testbed  MOEs  for  Test  Series  2 
would  measure  the  needed  information  for  the  C^  evaluation  objectives.  MCES  termed 
these  type  of  measures  as  Measures  of  Performance  (MOP)  since  they  were  measures 
of  the  air  defense  functions.  There  is  obvious  disagreement  in  methodologies  as  to 
what  to  name  the  different  set  of  measures.  This  was  not  an  important  detail  and  did 
not  cause  too  much  difficulty. 


3.  Functional  Measures 


Major  Gandee  s  approach  was  to  utilize  the  c’  functions  as  the  focal  point 
for  deriving  measures.  If  this  approach  is  to  be  strictly  followed,  then  each  function 
would  have  at  least  one  corresponding  probability  measure  plus  time  and  distance 
distribution  measures.  The  measures  of  probabilities  and  time  and  distance  ranges  for 
each  function  should  be  the  minimum  measures  used  to  measure  these  functions. 
Major  Gandee  proposed  these  measures  and  highlighted  that  all  the  functions  should 
have  corresponding  measures.  Table  4  lists  the  measures  that  should  be  included  for 
the  functions. 


TABLE  4 

MCES  FUNCTIONS  AND  MEASURES 

FUNCTION 

DETECT  (D) 

P( Detect), 

Tame  and  Distance 

TRACK  (T) 

P( Track) , 

Time,  and  Distance 

IDENTIFY  (ID) 

P( Identify) , 

Time  and  Distance 

ASSESS  THREAT  (TA) 

P( Assess  Threat), 

Time  and  Distance 

ASSIGN  WEAPON  (WA) 

P( Assign  Weapon), 

Time  and  Distance 

ALLOCATE  WEAPON  (AW) 

P( Allocate  Weapon), 

Time  and  Distance 

CONTROL  (C) 

P( Control), 

Time  and  Distance 

Most  of  these  measures  are  listed  in  the  test  design  plan  for  Test  Series  2.  Major  Grey 
of  the  1FFN  Testbed  does  state  that  the  data  for  all  these  measures  will  be  available 
but  some  were  not  considered  for  Test  Series  2.  Some  of  these  functions  will  be 
measured  indirectly  and  the  functional  measures  may  be  incorporated  in  later  tests  if 
the  results  of  Test  Series  2  reveals  that  they  are  needed. 
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4.  Distributed  Functions  and  Measures 


Major  Gandee  thought  that  the  measures  for  the  XTELL  and  I  NIL EL 
processes  must  be  used  to  effectively  evaluate  distributed  C~  systems.  If  this  is  indeed 
the  case  then  each  function  should  have  at  least  one  corresponding  measure  of 
performance.  Examples  of  these  are  shown  in  Table  5  .  Most  of  these  measures  were 
used  by  the  IFFN  Testbed.  The  IFFN  Testbed  will  use  their  measures  of  P(Pass). 
P(Res),  P(Trans),  and  P(Amp)  to  measure  the  indirect  ID  information  flow  which  will 
indirectly  measure  some  of  the  coordination  and  intelligence  functions.  These  IFFN 
measures  and  definitions  are  listed  for  review  and  comparison. 

•  P(Pass)  Probability  that  a  passed  ID  is  correct. 


P(Res)  Probability  chat  an  ID  conflict  is  resolved  while  the  aircraft  is  still  in 
the  weapon  system's  measurement  volume. 


TfTrans)  Distribution  of  times  elapsed  between  receipt  and  retransmission  of 
ID  information  bv  a  C2  node. 


P(Amp)  Probability  that  an  ID  includes  track  amplification  information. 


TABLE  5 


DISTRIBUTED  FUNCTIONS  AND  MOPS 


FUNCTION 


MEASURES 


CROSSTELL  (XTELL) 


P( Correct  Fusion) 


Track  Correlation  (TC) 


Timeliness,  accuracy 


ID  Conflict 
Resolution  ( IDR) 


Timeliness,  accuracy 


INTELLIGENCE  ( INTELL) 


PfTarget  Engagement,  given 
I ID  was  used) 

P( Target  Engagement,  given 
I ID  was  not  used) 


EXECUTION  LEVEL 
C2  PROCESS 


P( Correct  ID  prior  to 
engagement) 


iw 


5.  Operational  versus  Design/Qualitative  Measures 

While  most  evaluations  of  systems  center  around  design  qualitative 
measures  such  as  flexibility,  surviveability,  availability,  etc.,  the  IFFN  Testbed  used  an 
approach  much  like  the  MCES  methodology  in  that  the  functions  of  the  system  were 
first  studied  before  equipment  and  personnel  problems  were  considered.  These 
measures  are  sometimes  refered  to  as  operational  measures.  Examples  of  operational 
measures  for  distributed  C“  systems  are: 

•  CAPACITY 

•  ACCURACY 

•  RESPONSE  TIME 

•  AVAILABILITY 

•  THRU  PUT 

The  IFFN  Testbed  did  not  continue  their  evaluation  to  include  the  design  qualitative 
measures  that  are  definitely  needed  to  determine  the  utility  of  a  particular  competing 
C“  system.  Major  Grey  did  state  that  design  type  measures  would  be  incorporated  in 
later  tests  and  that  the  data  needed  for  these  types  of  measures  was  readily  available 
even  for  Test  Series  2  if  the  need  arose  or  if  a  higher  authority  requested  that 
information.  Examples  of  design  and  quality  measures  are: 

•  EFFICIENCY 

•  RELIABILITY 

•  SURVIVABILITY 

•  USEABILITY 

•  CORRECTNESS 

•  MAINTAINABLE 

•  VERIFIBIL1TY 

•  EXPANDABILITY 

•  FLEXIBILITY 

•  INTEROPERABILITY 

•  PORTABILITY 

•  REUSABILITY 

•  ROBUSTNESS 

•  EVOLV  ABILITY 

Another  analyst,  Leslie  Gollidav,  listed  a  set  of  generic  air  defense  measures. 
Table  6  [Ref.  15:  p.  788),  which  are  similar  to  what  the  MCES  application  had 


determined.  These  measures  are  more  general  but  the  specific  measures  could  be 
derived  from  the  general  measures.  Again,  the  list  included  function  measures  as  well 
as  design  qualitative  measures. 


TABLE  6 

C2 

MEASURES  FOR  AIR  DEFENSE 

MOP 

DEFINITION 

Alerting 

Capability 

Measures  capability  of  providing 
gross  positional  data  on  an  aircraft 
at  extended  ranges. 

Cueing 

Capability 

Measures  the  process  of  providing 
specific  and  timely  positional 
data  with  tentative  identification 
of  an  aircraft  within  a  designated 
range  of  a  unit. 

Weapons  Control 

Information 

Capability 

Measures  capability  to  provide 
weapons  control  order  (wCOs) 
and  rules  of  engagement  (ROEs). 

Airspace 

Management 

Capability 

Measures  capability  to  provide 
avoidance  of  engagement  of 
friendly  fixed  ana  rotary 
aircraft. 

MOEs 

INTEROPERABILITY 

RELIABILITY 

MAINTAINABILITY 

FLEXIBILITY 

USEABILITY 

SURVIVABILITY 

6.  Resource  Conservation  and  Reallocation  Measures 

Major  Gandee  suggested  possible  efficiency  and  coordination  measures  which 
would  reflect  the  probability  of  a  target  being  engaged  by  more  than  one  unit  due  to 
lack  of  coordination.  A  network  may  increase  an  individual  unit's  identification 
capabilities  by  supplying  more  complete  ID  information  from  other  units  which 


74 


formulate  the  ID  information.  Major  Gandee  suggested  that  an  ID  accuracy  measure 
could  be  used  to  compare  the  accuracy  of  an  ID  formulated  at  an  organic  unit  and  the 
ID  information  which  passes  over  the  network  to  other  units  as  a  system  ID.  Other 
possible  measures  suggested  by  Major  Gandee  were  fusing  measures  that  measured  the 
ability  of  units  to  accept  and  fuse  system  ID  information  with  its  own  organic  ID 
information  to  improve  the  ID  accuracy  in  time  to  use  it  effectively.  However,  the 
suggested  measures  of  weapon  allocation  efficiency,  unit  ID  coordination,  ID  accuracy, 
and  ID  fusing  ability  were  not  used  by  the  IFFN  Testbed.  Some  of  these  attributes 
could  be  measured  indirectly  using  some  of  the  other  IFFN  Testbed  measures  such  as 
timeliness  and  the  probability  of  correctly  identifying  an  object  to  measure  ID 
information  fusing  ability. 

Determining  which  competing  consumes  less  missile  resources  is  very 
important  considering  the  cost  and  shortage  of  modem  missiles.  Also,  in  a  dynamic 
environment,  there  will  be  instances  when  missile  resources  will  have  to  be  reallocated 
due  to  prior  destruction  of  the  target,  previous  allocation  to  another  weapon,  higher 
priority  targets  in  the  area  or  the  targets  flies  out  of  range.  Measures  of  efficiency  and 
reallocation  ability  seemed  to  be  crucial  differences  in  the  competing  C~  systems. 
Again.  Major  Gandee  suggested  such  measures  but  they  were  not  all  incorporated  into 
the  current  Test  Series  2  plan.  Research  by  Alphatec,  Inc.  with  Petri  nets  also 
suggested  reallocation  measures  for  the  IFFN  Testbed  [Ref.  14]. 

G,  DATA  GENERATION  AND  TESTBED  DESIGN 

Major  Gandee  s  proposed  application  of  MCES  to  the  IFFN  basically  stopped  at 
Module  5,  Specification  of  Measures  due  to  time  constraints  and  the  delay  of  Test 
Senes  2  execution.  As  a  result,  the  MCES  data  generation  and  aggregation  and 
interpretation  modules  were  not  evaluated  for  Test  Series  2  by  Major  Gandee. 

The  current  MCES  does  incorporate  a  experimental  design  methodology  for 
building  testbeds  to  generate  data.  A  methodology,  Systems  Effectiveness  Analysis 
(SEA),  was  introduced  by  A.  H.  Levis  and  P.  Derskin  [Ref.  16]  for  evaluating  large 
scale  systems  such  as  testbeds  and  ultimately  integrated  into  the  MCES  methodology. 
To  produce  valid  data,  a  testbed  must  be  credible  and  SEA  was  developed  to  assist  in 
validating  testbeds.  SEA  was  also  developed  to  determine  the  minimum  number  of 
experiments  needed  to  evaluate  the  system  and  to  formulate  a  optimal  sequence  of 
improvements  areas  for  the  competing  configurations.  Once  the  testbed  was 
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determined  to  he  credible,  experiments  could  be  run  that  would  determine  the  optimal 
effectiveness  of  each  alternative  O  system. 

One  of  Dr.  Levis'  students,  Phillipe  Martain,  demonstrated  how  SEA  could  be 
applied  to  the  IFFN  Testbed.  First  in  the  SEA  process  was  the  determination  of  the 
smallest  number  of  experiments  that  were  needed  to  be  run  to  evaluate  the 
effectiveness  of  the  testbed  system.  A  simplified  mathematical  model  utilizing 
Lanchester  type  combat  models  was  developed  to  determine  the  minimum  number  of 
experiments  required  to  evaluate  the  testbed.  The  testbed  experimental  results  and  the 
mathematical  model  results  could  also  be  compared  to  insure  similar  results.  In  a 
second  phase,  a  system  planning  procedure  was  used  to  select  the  best  evolution  path 
for  the  testbed  configurations  from  a  fixed  set  of  improvements.  SEA  can  be  then  used 
in  the  last  MCES  module  of  Data  Aggregation  and  Interpretation  to  make  adjustments 
to  the  testbed  experiments.  (Ref.  17] 

H.  OVERLAPPING  OF  PROCEDURES  AND  TOOLS 

The  IFFN  Testbed  has  been  working  on  its  air  air  defense  problem  for  a 
number  of  years  and  through  its  iterative  process  evolved  to  a  solution  that  was  close 
to  the  MCES  solution.  The  IFFN  Testbed  started  without  the  aid  of  MCES  and  some 
of  the  applied  MCES  methodology  overlapped  with  the  previous  methods  used  by  the 
IFFN  Joint  Test  Force.  However,  in  most  cases  both  methodologies  resulted  in  the 
same  general  results.  The  IFFN  Testbed  did  make  changes  after  the  MCES 
application,  but  it  is  not  clear  if  these  changes  will  have  a  major  impact  on  Test  Series 
2  since  the  test  has  not  been  completed. 

MCES  does  integrate  and  imbed  current  evaluation  tools.  MCES  does  not 
preclude  these  tools  and  in  fact  uses  them  to  obtain  a  better  solution  to  the  problem. 
Both  the  IFFN  and  MCES  methods  came  to  the  same  general  conclusions,  however 
the  MCES  approach  appeared  to  be  more  complete.  The  MCES  methodology- 
attempts  to  standardize  the  analysis  by  providing  a  structured  template  to  assist  the 
analysts  in  their  evaluations. 

1 .  Problem  Formulation  and  Bounding 

The  problem  formulation  and  bounding  of  the  C2  system  were  almost 
identical.  Most  system  analysis  methodologies  start  out  in  this  same  manner. 


2.  Functional  Analysis 

Other  system  analysis  approaches  use  the  basic  input,  process,  and  output 
approach  to  describing  the  C"  system.  Functional  and  process  analysis  are  being  used 
often  in  the  software  engineering  environment.  These  approaches  are  similar  to  C. 
West  Churchman's  system  process  of  input,  process,  and  output  as  explained  by 
Schoderbek,  et  al.  [Ref.  18:  pp.  8-29j.  It  is  interesting  to  note  that  major  methodology 
revisions  occured  when  analysts  attempted  to  automate  systems  because  they  needed  to 
precisely  describe  and  recreate  the  functioning  of  the  manual  system. 

3.  Model  or  Architecture  Building 

The  system  analysis  approach  of  utilizing  the  functions  of  the  system  to  build 
a  systems  model  has  been  used  in  a  number  of  previous  evaluations.  Other  researchers 
have  added  methods  of  modeling  that  can  be  integrated  into  the  MCES  methodology. 
A  specific  example  already  referenced  is  Systems  Effectiveness  Analysis  (SEA).  Dr. 
Levis  has  conducted  stimulating  research  in  this  bottom-up  approach  and  has 
integrated  it  into  the  MCES  methodology  [Ref.  19].  The  SEA  methodology  insures 
that  the  simulation  can  simulate  the  whole  range  or  limits  of  the  interactions  between 
the  variables  instead  of  a  smaller  subset  of  the  interaction  range  or  limit.  This  can  be 
accomplished  by  taking  established  measures  and  and  determining  their  minimum  and 
maximum  ranges.  The  other  quantitative  methods  of  SEA  can  also  be  applied  to  the 
IFFN  Testbed. 

Petri  Nets  have  been  used  by  researchers  and  analysts  to  mathematically 
model  the  different  structures  of  information  flow  in  the  C"  architecture  in  the  form  of 
off  and  on  states  which  can  be  used  to  describe  a  C^  architecture  with  unique 
structures.  Alphatec,  Inc.  [Ref.  14]  conducted  a  study  of  the  IFFN  Testbed  and 
constructed  a  number  of  different  level  petri  nets  to  determine  what  measures  were 
needed  to  measure  what  kind  and  how  much  information  flowed  between  all  the  nodes. 
Basically,  each  connection  between  the  nodes  was  an  opportunity  to  measure 
information  flow. 

4.  IFFN  and  MCES  Measure  Specification 

The  IFFN  Test  Force  used  its  issues  to  formulate  their  original  measures. 
However,  there  were  no  major  deficiencies  in  the  test  design  of  Test  Series  1  which  was 
completed  pnor  to  the  MCES  application.  There  was  prior  research  conducted  by  the 
Institute  for  Defense  Analysis  [Ref.  13]  and  Alphatec  [Ref.  14]  concerning  the 
evaluation  of  the  IFFN  competing  systems  and  they  confirmed  the  measures 
derived  by  the  MCES  application. 
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I.  COMPARISON  SUMMARY 


The  IFFN  Testbed  has  already  taken  advantage  of  the  good  ideas  generated  by 
the  MCES  application  and  the  Test  Series  2  plan  has  incorporated  some  of  the  MCES 
concepts.  Although  each  methodology  used  different  and  similar  tools  to  evaluate  the 
competing  C“  systems,  both  approaches  came  to  the  same  general  conclusions.  The 
question  of  whether  the  amount  of  time  needed  to  document  all  possible  interactions  in 
the  MCES  is  really  needed  is  still  unanswered.  However  after  utilizing  the  MCES 
approach  of  analyzing  the  physical  entites,  organizational  structure,  and  C^  functions 
in  a  systematic  methodology,  the  IFFN  Test  Force  discovered  a  number  of  important 
measures  that  they  had  not  focussed  on  earlier  in  the  test  design  process  for  Test  Series 
2.  Only  with  more  testing  and  comparisons  will  the  true  value  of  the  MCES  approach 
to  the  IFFN  Testbed  be  known.  Conclusions  and  recommendations  concerning  both 
the  IFFN  Testbed  and  MCES  are  included  in  Chapter  VI  of  this  thesis. 


VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  I FFN  CONCLUSIONS 

It  is  clearly  evident  that  the  I  FFN  Testbed  has  made  some  progress  in  solving 
some  of  the  I  FFN  air  defense  problems.  The  problem  is  definitely  complex  and  the 
I  FFN  Testbed  has  understandably  committed  large  amount  of  resources  to  the 
problem.  The  Testbed  is  a  good  concept  for  an  experimental  design  to  test  alternative 
C“  systems  since  all  of  the  alternative  systems  can  not  be  tested  using  actual 
equipment  due  to  resource  constraints  or  present  configuration  limitations.  The 
I  FFN  Test  Force  was  careful  to  insure  that  the  testbed  was  credible.  Only  one  test 
series  has  been  completed  with  Test  Series  2  to  begin  in  March  1987.  The  I  FFN  Joint 
Testbed  has  provided  an  excellent  opportunity  to  test  and  refine  MCES.  The  IFFN 
Testbed  is  still  valuable  as  a  suitable  data  generator  for  further  evaluatation  and 
refinement  of  MCES. 

The  IFFN  Testbed  has  already  taken  advantage  of  the  good  results  generated  by 
the  MCES  application  and  the  Test  Series  2  plan  has  incorporated  some  of  the  MCES 
ideas.  After  utilizing  the  MCES  approach  of  analyzing  the  physical  entites. 
organizational  structure,  and  C^  functions  in  a  systematic  methodology,  the  IFFN  Test 
Force  discovered  a  number  of  important  measures  they  had  not  focussed  on  earlier  in 
the  test  design  process  for  Test  Series  2.  The  test  design  issues  and  measures  were 
modified  to  accommodate  the  newly  found  distributed  relationships  between  C~  nodes 
that  were  originally  not  formulated  by  the  IFFN  Joint  Test  Force. 

B.  IFFN  RECOMMENDATIONS 

An  additional  measure  approach  would  be  to  utilize  both  operational  and 
equipment  design  quality  measures.  The  functional  measures  are  operational  measures 
and  the  design  quality  measures  are  more  machine  and  resource  oriented.  The  IFFN 
Testbed  seems  to  have  focused  its  measures  on  operational  or  functional  measures  and 
has  not  taken  full  advantage  of  available  and  possibly  critical  design  qualitative 
measures  such  as  resource  efficiency. 

The  IFFN  Testbed  should  consider  Major  Gandee's  recommendations  on  the 
additional  measures  of  performance  and  effectiveness,  particularly  the  measures  of 
coordination  and  resource  allocation.  Resource  allocation,  connectivity,  availability, 
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surviveability.  sustainability,  and  flexibility  data  appears  to  be  available  from  the  data 
collection  points  in  the  simulations.  The  measures  for  the  XTELL  and  INTEEL 
processes  must  be  used  to  effectively  evaluate  distributed  C“  systems. 

C.  MCES  APPLICATION  CONCLUSIONS 

The  IFFN  Testbed  has  been  working  on  its  air  defense  C^  problem  for  a  number 
of  years  and  through  its  iterative  process  evolved  to  a  solution  that  was  close  to  the 
same  results.  There  is  a  learning  curve  associated  with  applying  any  new  methodology 
which  was  quite  evident  in  this  IFFN  application.  During  the  MCES  application  to 
the  IFFN  Testbed,  there  was  overlap  of  the  evaluation  tools  used  by  the  IFFN  Test 
Force.  Some  of  the  tools  used  prior  to  the  MCES  application  to  the  IFFN  Testbed 
were  included  in  the  MCES  methodology.  MCES  does  not  preclude  these  tools  and  in 
fact  uses  them  to  aid  in  the  evaluation  to  obtain  the  best  results.  This  MCES 
application  was  a  good  start  in  the  evolving  of  a  generic  standard  C"  system  evaluation 
method.  The  MCES  approach  of  systematically  outlining  the  physical  entities, 
structure,  and  C"  process  functions  insured  that  all  areas  were  covered.  Major 
Gandee's  proposed  application  was  used  by  the  IFFN  Operational  Analysis  section  to 
better  understand  their  air  defense  C^  problem.  The  MCES  approach  seemed  to  be 
more  detailed  and  complete.  New  relationships  were  uncovered  by  Major  Gandee 
resulting  in  the  addition  of  new  issues  and  measures.  MCES  has  definitely  helped  in 
highlighting  the  functional  measures  that  have  been  overlooked  in  previous  C ~ 
evaluations.  MCES  did  assist  the  IFFN  Testbed. 

1.  Integration  of  System  Elements  and  Functions  Module 

Major  Gandee  uncovered  the  need  for  the  additional  module  4.  Integration  of 
System  Elements  and  Functions,  that  was  not  originally  conceived  as  a  part  of  the 
MCES  modules.  A  model  or  architecture  was  needed  to  establish  a  baseline  with 
alternative  structures  to  represent  the  competing  C^  systems.  The  system  must  be 
understood  before  attempting  to  measure  its  utility  and  uncover  the  variables  are 
responsible  for  significant  differences.  By  actually  using  MCES.  some  problems 
surfaced  which  ultimately  resulted  in  refinements  to  the  MCES  methodology. 

2.  Distributed  C^  Process  Functions 

The  distributed  functions  are  required  to  characterize  the  coordination  and 
intelligence  sharing  of  distributed  C‘  systems.  There  are  different  levels  of  these 
distributed  functions  and  the  C“  structures  can  become  very  complex.  MCES  s 
distributed  functions  assist  in  describing  these  complexities. 
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3.  Solid  Evaluation  Tool 

MCES  appears  to  be  a  viable  C“  evaluation  template  of  current  and  evolving 
tools  based  on  solid  C“  theory.  MCES  has  provided  the  analyst  community  with  a 
theoretical  framework  for  determining  the  utility  of  a  C“  system. 

4.  MCES  Application  to  Additional  IFFN  Testbed  Test  Series 

The  IFFN  Testbed  did  revise  their  Test  Series  2  plan  to  incorporate  most  of 
Major  Gandee  s  results  of  applying  MCES.  Major  Grey  has  also  utilized  some  of  the 
concepts  of  MCES  to  formulate  the  design  plan  for  Test  Series  3.  As  more  test  series 
are  completed,  a  more  through  evaluation  of  MCES  can  be  made. 

D.  MCES  RECOMMENDATIONS 

1.  Continued  MCES  Application  to  the  IFFN  Testbed 

The  application  of  MCES  to  the  IFFN  Testbed  should  be  continued  on  Test 
Series  2  and  following  tests.  Due  to  time  constraints  and  the  delay  of  the  Test  Series  2 
execution,  the  final  results  were  not  available  for  analysis  in  this  thesis  as  was  originally 
envisioned.  The  areas  of  actual  data  generation  and  aggregation  of  measures  and 
interpretation  for  Test  Series  2  still  could  be  a  valuable  thesis  topic  for  a  follow-on 
project.  This  follow-on  project  could  document  and  evaluate  the  success  of  the 
previous  implementation  of  MCES  and  observe  new  implementations.  A  comparison 
of  Test  Series  1  with  Test  Series  2  results  might  reveal  the  differences  in  the  approach 
and  the  results  with  the  caveat  that  Test  Series  2  was  a  slightly  different  simulation 
than  Test  Series  1. 

2.  Further  MCES  Testing  and  Refinement 

More  beginning  to  end  C^  applications  of  MCES  should  be  conducted  to 
continue  the  evolution  of  the  MCES  methodology. 

3.  Integration  of  More  C*  Evaluation  Tools 

MCES  does  integrate  a  number  of  tools  and  could  still  integrate  more  while 
maintaining  its  solid  theoretical  base.  The  MCES  approach  is  a  top-down  systems 
approach  with  certain  advantages  and  disadvantages.  Utilizing  Dr.  Levis'  experimental 
design  and  bottom-up  approach.  Systems  Effectiveness  Analysis  (SEA),  would  benefit 
the  MCES  toolbox.  Petri  nets  show  promise  of  being  a  good  analyst  tool  lor 
evaluating  information  flow.  Other  system  evaluation  methods  should  also  researched 
for  addition  to  the  MCES  methodology. 


4.  Standard  MCES  Terminology  and  Definitions 

MCES  developers  must  decide  on  standard  terminology  and  definition*  to 
avoid  confusion.  Although  MCES  should  be  robust  and  flexible  to  analysts,  a  firm 
evaluation  theory  must  be  presented  in  simple,  standard  terminology. 

5.  Education  and  Dissemination 

There  is  a  learning  curve  associated  with  any  new  or  revised  methodology 
which  was  quite  evident  in  this  IFFN  application.  After  a  standard  terminology  is 
defined,  MCES  should  be  announced  and  advertised  to  the  C"  analyst  and  decision 
maker  community  because  of  its  sound  theoretical  background.  More  disclosure  of 
MCES  is  definitely  needed.  Since  MCF.S  does  incorporate  a  number  of  known  tools. 
MCES  should  be  advertised  as  a  structured  approach  to  utilize  C“  evaluation  methods 
and  tools. 
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